|
Replies:
24
-
Last Post:
Apr 29, 2006 5:25 AM
by: ian
|
|
|
Posts:
47
From:
US
Registered:
5/12/05
|
|
|
|
Google Summer of Code proposal - check it out...
Posted:
Apr 20, 2006 3:53 PM
|
|
Tools community folks,
Over at the Google Summer of Code proposal that OpenSolaris is doing, I'm proposing a "how to get more build parallelism on highly-parallel chips" project (or study or whatever you want to call it) . I've added a short blurb on this over at the wiki, located here:
http://www.genunix.org/wiki/index.php/Summer_of_Code#Exploiting_Parallelism_to_Optimize_Build_Performance
I think this is potentially both a performance project and a build/tools project, so I (presumptively, I know!) added the Tools Community to the Project Home list, hoping that the two communities might work together to sponsor/mentor a student.
If anybody would like to be a contact/mentor, please add yourself to the Wiki page over there!
And, if there's no interest at all, let me know, and I'll remove the Tools Community from the Project Home list....
Thanks,
Mike
|
|
|
Posts:
941
From:
Redwood City, CA
Registered:
3/9/05
|
|
|
|
Re: Google Summer of Code proposal - check it out...
Posted:
Apr 21, 2006 1:37 PM
in response to: mpogue
|
|
Quoth Mike Pogue on Thu, Apr 20, 2006 at 03:53:43PM -0700:
> Tools community folks,
>
> Over at the Google Summer of Code proposal that OpenSolaris is
> doing, I'm proposing a "how to get more build parallelism on
> highly-parallel chips" project (or study or whatever you want to
> call it) . I've added a short blurb on this over at the wiki,
> located here:
>
> http://www.genunix.org/wiki/index.php/Summer_of_Code#Exploiting_Parallelism_to_Optimize_Build_Performance
Have you heard of Electric Cloud?
David
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
47
From:
US
Registered:
5/12/05
|
|
|
|
Re: Google Summer of Code proposal - check it out...
Posted:
Apr 21, 2006 1:55 PM
in response to: bustos
|
|
I hadn't heard of it, but it looks pretty cool. (http://www.electric-cloud.com/)
I wonder how much speedup we could get from Electric Cloud running on a 32-thread Niagara box? (It seems to be designed more for clusters of boxes, rather than single boxes with many threads per box, but there could be a lot of leverage there).
Does anybody know how much this thing costs? (It might be like fine wine -- if you have to ask, it's more than you can afford. :-)
Mike
|
|
|
|
|
Posts:
941
From:
Redwood City, CA
Registered:
3/9/05
|
|
|
|
Re: Re: Google Summer of Code proposal - check it
out...
Posted:
Apr 21, 2006 3:12 PM
in response to: mpogue
|
|
Quoth Mike Pogue on Fri, Apr 21, 2006 at 01:55:58PM -0700:
> I hadn't heard of it, but it looks pretty cool.
> (http://www.electric-cloud.com/)
>
> I wonder how much speedup we could get from Electric Cloud running on
> a 32-thread Niagara box? (It seems to be designed more for clusters
> of boxes, rather than single boxes with many threads per box, but
> there could be a lot of leverage there).
As Danek notes, the problem with using it for Solaris is that it doesn't
support our Makefiles. We've already invested parallelization effort in
utilizing Studio's make, which has unfortunately rendered us ineligible
for Eletric Cloud's automatic analysis. So the project effort might be
better spent writing gmake Makefiles and asking Electric Cloud for
a trial.
David
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Stephen Hahn
sch@eng.sun.com
|
|
|
|
Re: Re: Google Summer of Code proposal - check it
out...
Posted:
Apr 21, 2006 3:34 PM
in response to: bustos
|
|
* David Bustos <David dot Bustos at sun dot com> [2006-04-21 15:12]:
> Quoth Mike Pogue on Fri, Apr 21, 2006 at 01:55:58PM -0700:
> > I hadn't heard of it, but it looks pretty cool.
> > (http://www.electric-cloud.com/)
> >
> > I wonder how much speedup we could get from Electric Cloud running on
> > a 32-thread Niagara box? (It seems to be designed more for clusters
> > of boxes, rather than single boxes with many threads per box, but
> > there could be a lot of leverage there).
>
> As Danek notes, the problem with using it for Solaris is that it doesn't
> support our Makefiles. We've already invested parallelization effort in
> utilizing Studio's make, which has unfortunately rendered us ineligible
> for Eletric Cloud's automatic analysis. So the project effort might be
> better spent writing gmake Makefiles and asking Electric Cloud for
> a trial.
We actually know (or at least knew and could know again) a lot about
the build. Alan wrote a paper on the ON build sometime ago for the
SunTech Conference in 2001. See
http://mediacast.sun.com/share/sch/d-exacct-builds-suntech-2001.pdf
for details.
(A number of aspects have changed since this paper was written; for
instance, the task's ancestor is now part of the default task record
and PRIV_PROC_TASKID can be assigned to a user, meaning that they can
call settaskid(2) without being root.)
- Stephen
--
Stephen Hahn, PhD Solaris Kernel Development, Sun Microsystems
stephen dot hahn at sun dot com http://blogs.sun.com/sch/
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
53
From:
Registered:
3/9/05
|
|
|
|
Re: Re: Google Summer of Code proposal - check it
Posted:
Apr 22, 2006 2:41 PM
in response to: bustos
|
|
> ... We've already invested parallelization effort in
> utilizing Studio's make, which has unfortunately
> rendered us ineligible for Electric Cloud's automatic analysis.
It sounds very strange to me. What particularly could be
changed in makefiles to improve the parallelization utilizing
Sun Studio dmake, that can make these makefiles unusable
for gmake? There is only one special target, and its name
is the same for both (dmake and gmake):
.PARALLEL: t1 t2 t3
That's it, nothing else.
There are several ways to block parallelization, and some of
them are slightly different:
.NO_PARALLEL: t3 t4 # Sun Studio dmake
.NOTPARALLEL: t3 t4 # GNU make
all: t1 t2 t3 .WAIT t4 t5 # Sun Studio dmake
# GNU make does not support .WAIT yet, but it will soon (probably)
The recommendation is to properly specify the dependencies instead
of using .WAIT:
t4: t1 t2 t3
t5: t1 t2 t3
This way to block the parallelization will work for any "make".
> So the project effort might be better spent writing gmake
> Makefiles and asking Electric Cloud for a trial.
The main difference between Sun make and GNU make is not
in parallelization (and Sun Studio dmake is probably better than
GNU make and Electric Cloud in parallelization). There are
many serious differences in syntax between Sun make and
GNU make, and that's the main problem why GNU make cannot
understand Solaris makefiles.
BTW, some time ago we investigated the level of parallelization
in Solaris ON build, and we found out that it is very good. On a
4 CPU system, the coefficient was 3.6. We used Sun Studio dmake,
and Solaris 9 sources.
Thanks,
Nik
|
|
|
|
|
Posts:
941
From:
Redwood City, CA
Registered:
3/9/05
|
|
|
|
Re: Re: Re: Google Summer of Code proposal - check it
Posted:
Apr 22, 2006 5:20 PM
in response to: nikm
|
|
Quoth Nikolay Molchanov on Sat, Apr 22, 2006 at 02:41:30PM -0700:
> > ... We've already invested parallelization effort in
> > utilizing Studio's make, which has unfortunately
> > rendered us ineligible for Electric Cloud's automatic analysis.
>
> It sounds very strange to me. What particularly could be
> changed in makefiles to improve the parallelization utilizing
> Sun Studio dmake, that can make these makefiles unusable
> for gmake? There is only one special target, and its name
> is the same for both (dmake and gmake):
Then I was mistaken. It's the use of general Sun-make-specific
constructs (like conditional macro assignments) rather than
Sun-make-specific parallelization ones.
> > So the project effort might be better spent writing gmake
> > Makefiles and asking Electric Cloud for a trial.
>
> The main difference between Sun make and GNU make is not
> in parallelization (and Sun Studio dmake is probably better than
> GNU make and Electric Cloud in parallelization).
Perhaps you misunderstand Electric Cloud. It provides benefit not
through a better make (though maybe that's part of it), but by
filesystem-level dependency analysis, distribution, and file caching.
> There are
> many serious differences in syntax between Sun make and
> GNU make, and that's the main problem why GNU make cannot
> understand Solaris makefiles.
Right.
> BTW, some time ago we investigated the level of parallelization
> in Solaris ON build, and we found out that it is very good. On a
> 4 CPU system, the coefficient was 3.6. We used Sun Studio dmake,
> and Solaris 9 sources.
Right. This is why I haven't brought this up before. Given that we
already parallelize pretty well, and we so widely use Sun-make-specific
features, it seems unlikely that converting the Makefiles will be worth
the effort.
David
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
53
From:
Registered:
3/9/05
|
|
|
|
Re: Re: Re: Google Summer of Code proposal - check it
Posted:
Apr 23, 2006 8:57 AM
in response to: bustos
|
|
> Perhaps you misunderstand Electric Cloud.
> It provides benefit not through a better make (though
> maybe that's part of it), but by filesystem-level dependency
> analysis, distribution,and file caching.
Oh, yes, I think you are right. BTW, it is interesting to estimate
what improvement we can get from file caching. Sun Studio 11
compilers have a new feature: "preprocessed include files". This
is some kind of caching, and we can try to use it. Another way is
to have /usr/include in memory (f.e. copy this directory to /tmp
and mount as /usr/include). What else makes sense to cache?
Thanks,
Nik
|
|
|
|
|
Posts:
818
From:
San Francisco CA US
Registered:
3/9/05
|
|
|
|
Re: Re: Re: Re: Google Summer of Code proposal -
check it
Posted:
Apr 23, 2006 10:34 AM
in response to: nikm
|
|
On Sun, Apr 23, 2006 at 08:57:47AM -0700, Nikolay Molchanov wrote:
> Oh, yes, I think you are right. BTW, it is interesting to estimate
> what improvement we can get from file caching. Sun Studio 11
> compilers have a new feature: "preprocessed include files". This
> is some kind of caching, and we can try to use it. Another way is
This sounds a lot like ccache[0], which caches compiler output and
products based on a hash of the compilation command line and the
preprocessor output. But this has little to do with parallelism.
[0] http://ccache.samba.org/
--
Keith M Wesolowski "Sir, we're surrounded!"
Solaris Kernel Team "Excellent; we can attack in any direction!"
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
53
From:
Registered:
3/9/05
|
|
|
|
Re: Re: Re: Re: Google Summer of Code proposal -
Posted:
Apr 23, 2006 1:32 PM
in response to: wesolows
|
|
> > Sun Studio 11 compilers have a new feature: "preprocessed include
> > files". This is some kind of caching, and we can try to use it.
>
> This sounds a lot like ccache[0], which caches compiler output and
> products based on a hash of the compilation command line and the
> preprocessor output. But this has little to do with parallelism.
Completely agree. Though there is one key point: if there is a "bottle
neck" (a target takes long time, and all other targets cannot start
while this one is not done), any improvement in this particular target
will help. So, it seems to make sense to try to use file caching to
minimze the time lost in "bottlenecks".
Thanks,
Nik
|
|
|
|
|
Posts:
1,710
From:
NZ
Registered:
4/27/05
|
|
|
|
Re: Re: Re: Re: Re: Google Summer of Code proposal -
Posted:
Apr 23, 2006 2:36 PM
in response to: nikm
|
|
Nikolay Molchanov wrote:
>>>Sun Studio 11 compilers have a new feature: "preprocessed include
>>>files". This is some kind of caching, and we can try to use it.
>>>
>>>
>>This sounds a lot like ccache[0], which caches compiler output and
>>products based on a hash of the compilation command line and the
>>preprocessor output. But this has little to do with parallelism.
>>
>>
>
>Completely agree. Though there is one key point: if there is a "bottle
>neck" (a target takes long time, and all other targets cannot start
>while this one is not done), any improvement in this particular target
>will help. So, it seems to make sense to try to use file caching to
>minimze the time lost in "bottlenecks".
>
>
>
As I've suggested before, the best way I can see to improve build times
is to "flatten out" the build. One way to do this is to create higher
level makefiles with targets from several directories. This reduces the
number of bottlenecks.
This is how I structure my own code and I can maintain a steady number
of jobs over several build machines this way.
This will also help to bring distributed building to ON.
Ian
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,218
From:
Registered:
3/9/05
|
|
|
|
Re: Re: Re: Re: Re: Google Summer of Code proposal -
Posted:
Apr 24, 2006 7:19 AM
in response to: ian
|
|
Ian Collins wrote:
> As I've suggested before, the best way I can see to improve build times
> is to "flatten out" the build. One way to do this is to create higher
> level makefiles with targets from several directories. This reduces the
> number of bottlenecks.
>
> This is how I structure my own code and I can maintain a steady number
> of jobs over several build machines this way.
>
> This will also help to bring distributed building to ON.
As Stephen Hahn has already pointed out I did a significant amount of
work in this area in the past. I too incorrectly assumed that the thing
slowing down builds was lack of parallelism in the makefiles, and I was
wrong as well. The actual cause was unnecessary /bin/sh invocations
which were chewing up vast amounts of CPU, and I suspect that there is
still work to be done in that area. This was directly contrary to my
expectations when I started the work, I thought the problems would be
primarily related to poor parallelism in the build.
Please read the paper.
http://mediacast.sun.com/share/sch/d-exacct-builds-suntech-2001.pdf
Moral of the story: Instrument and measure before suggesting solutions,
otherwise you are probably fixing the wrong thing.
--
Alan Burlison
--
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
47
From:
US
Registered:
5/12/05
|
|
|
|
Re: Re: Re: Re: Re: Google Summer of Code proposal -
Posted:
Apr 24, 2006 1:47 PM
in response to: alanbur
|
|
Based on the comments/suggestions so far (thanks all!), I've broadened the project description. I'm intentionally keeping it fairly open-ended, because a SoC student may have even more ideas about how to make the OpenSolaris build go faster -- ideas which we would never think of.
The main thing for me is to make some highly parallel hardware available, then turn somebody loose on it to see what happens!
|
|
|
|
|
Posts:
1,710
From:
NZ
Registered:
4/27/05
|
|
|
|
Re: Re: Re: Re: Re: Google Summer of Code proposal -
Posted:
Apr 24, 2006 4:06 PM
in response to: alanbur
|
|
Alan Burlison wrote:
> Ian Collins wrote:
>
>> As I've suggested before, the best way I can see to improve build times
>> is to "flatten out" the build. One way to do this is to create higher
>> level makefiles with targets from several directories. This reduces the
>> number of bottlenecks.
>>
>> This is how I structure my own code and I can maintain a steady number
>> of jobs over several build machines this way.
>>
>> This will also help to bring distributed building to ON.
>
>
> As Stephen Hahn has already pointed out I did a significant amount of
> work in this area in the past. I too incorrectly assumed that the
> thing slowing down builds was lack of parallelism in the makefiles,
> and I was wrong as well. The actual cause was unnecessary /bin/sh
> invocations which were chewing up vast amounts of CPU, and I suspect
> that there is still work to be done in that area. This was directly
> contrary to my expectations when I started the work, I thought the
> problems would be primarily related to poor parallelism in the build.
>
> Please read the paper.
>
> http://mediacast.sun.com/share/sch/d-exacct-builds-suntech-2001.pdf
>
> Moral of the story: Instrument and measure before suggesting
> solutions, otherwise you are probably fixing the wrong thing.
>
An interesting paper.
The conclusion I draw from it is that the biggest problem is the uneven
distribution of jobs throughout the build process. I would be
interested in seeing data for the number of instances of cc over time.
I have found it best to consider parallel/distributed builds from day
one and to continuously monitor builds for any bottlenecks. My process
is crude, simply track the number of jobs reported by dmake and the CPU
load of the build machines. I also periodically experiment with the
jobs/CPU ratio for my build servers as the nature of the source files
changes.
I have also found that splitting (or occasionally combining) source
files can have a significant impact.
Ian.
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,218
From:
Registered:
3/9/05
|
|
|
|
Re: Re: Re: Re: Re: Google Summer of Code proposal -
Posted:
Apr 24, 2006 4:14 PM
in response to: ian
|
|
Ian Collins wrote:
> The conclusion I draw from it is that the biggest problem is the uneven
> distribution of jobs throughout the build process. I would be
> interested in seeing data for the number of instances of cc over time.
>
> I have found it best to consider parallel/distributed builds from day
> one and to continuously monitor builds for any bottlenecks. My process
> is crude, simply track the number of jobs reported by dmake and the CPU
> load of the build machines. I also periodically experiment with the
> jobs/CPU ratio for my build servers as the nature of the source files
> changes.
>
> I have also found that splitting (or occasionally combining) source
> files can have a significant impact.
I still have the bits and pieces we used to do this analysis, there is
no reason why they couldn't be dusted off and reused. I'd be happy to help.
--
Alan Burlison
--
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,710
From:
NZ
Registered:
4/27/05
|
|
|
|
Re: Re: Re: Re: Re: Google Summer of Code proposal -
Posted:
Apr 24, 2006 4:46 PM
in response to: alanbur
|
|
Alan Burlison wrote:
> Ian Collins wrote:
>
>> The conclusion I draw from it is that the biggest problem is the uneven
>> distribution of jobs throughout the build process. I would be
>> interested in seeing data for the number of instances of cc over time.
>>
>> I have found it best to consider parallel/distributed builds from day
>> one and to continuously monitor builds for any bottlenecks. My process
>> is crude, simply track the number of jobs reported by dmake and the CPU
>> load of the build machines. I also periodically experiment with the
>> jobs/CPU ratio for my build servers as the nature of the source files
>> changes.
>>
>> I have also found that splitting (or occasionally combining) source
>> files can have a significant impact.
>
>
> I still have the bits and pieces we used to do this analysis, there is
> no reason why they couldn't be dusted off and reused. I'd be happy to
> help.
>
I'd be interested in that, I'd like to try your instrumentation on my
current project tree, which is built form one flat makefile.
Ian
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,218
From:
Registered:
3/9/05
|
|
|
|
Re: Re: Re: Re: Re: Google Summer of Code proposal -
Posted:
Apr 24, 2006 11:24 PM
in response to: ian
|
|
Ian Collins wrote:
> I'd be interested in that, I'd like to try your instrumentation on my
> current project tree, which is built form one flat makefile.
I'll make sure that it's clean of any encumbrances and get a copy out
there somewhere. As Stephen pointed out, a lot of the hacky bits aren't
needed any more as we now have LeastPriv. It was also set up to use
Oracle as a datastore, so switching to MySQL is probably in order as well.
--
Alan Burlison
--
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Rayson Ho
rayrayson@gmail.com
|
|
|
|
Re: Re: Re: Re: Re: Google Summer of Code proposal -
Posted:
Apr 28, 2006 9:35 PM
in response to: alanbur
|
|
Someone mentioned distributed builds with Electric Cloud, I am not
sure whether everyone read this or not:
10–20x Faster Software Builds, John Ousterhout, Electric Cloud, Inc.
http://www.usenix.org/events/usenix05/tech/slides/ouster.pdf
http://www.usenix.org/events/usenix05/tech/mp3/ousterhout.mp3
Also, the Sun GridEngine ( http://gridengine.sunsource.net ) didn't
get accepted in this year's SoC as a mentor, but if there is
interested with distributed builds, may be GridEngine's "qmake" is
something interesting to look at??
Or may be there can be a join GridEngine+OpenSolaris project for
SoC... hmm, may be...
Rayson
On 4/25/06, Alan Burlison <Alan dot Burlison at sun dot com> wrote:
> Ian Collins wrote:
>
> > I'd be interested in that, I'd like to try your instrumentation on my
> > current project tree, which is built form one flat makefile.
>
> I'll make sure that it's clean of any encumbrances and get a copy out
> there somewhere. As Stephen pointed out, a lot of the hacky bits aren't
> needed any more as we now have LeastPriv. It was also set up to use
> Oracle as a datastore, so switching to MySQL is probably in order as well.
>
> --
> Alan Burlison
> --
> _______________________________________________
> tools-discuss mailing list
> tools-discuss at opensolaris dot org
>
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,710
From:
NZ
Registered:
4/27/05
|
|
|
|
Re: Re: Re: Re: Re: Google Summer of Code proposal -
Posted:
Apr 29, 2006 5:25 AM
in response to: Rayson Ho
|
|
Rayson Ho wrote:
> Someone mentioned distributed builds with Electric Cloud, I am not
> sure whether everyone read this or not:
>
> 10–20x Faster Software Builds, John Ousterhout, Electric Cloud, Inc.
> http://www.usenix.org/events/usenix05/tech/slides/ouster.pdf
> http://www.usenix.org/events/usenix05/tech/mp3/ousterhout.mp3
>
Interesting paper, I've been using the same arguments to justify
Solaris+dmake as a development platform in preference to windows or
Linux for several years now.
Unfortunately I don't have enough nodes (4) to reproduce all of their
test results, but I have seen similar effects with makefile design.
A quick, unscientific comparison based on the charts of slide 27 yielded
the following results:
Apache - 4 nodes 2.5x faster than 1.
MySql5 - 4 nodes 2.1x faster than 1.
Mozilla requires gmake.
So better than distcc, not as good as electric cloud.
Ian
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,172
From:
US
Registered:
3/9/05
|
|
|
|
Re: Re: Re: Re: Google Summer of Code proposal -
check it
Posted:
Apr 24, 2006 2:00 PM
in response to: nikm
|
|
Nikolay Molchanov wrote:
>> Perhaps you misunderstand Electric Cloud.
>> It provides benefit not through a better make (though
>> maybe that's part of it), but by filesystem-level dependency
>> analysis, distribution,and file caching.
>
> Oh, yes, I think you are right. BTW, it is interesting to estimate
> what improvement we can get from file caching. Sun Studio 11
> compilers have a new feature: "preprocessed include files". This
> is some kind of caching, and we can try to use it. Another way is
> to have /usr/include in memory (f.e. copy this directory to /tmp
> and mount as /usr/include). What else makes sense to cache?
>
> Thanks,
> Nik
Oblig. caveat: ON doesn't include files from /usr/include
because it is the source of those include files ;-).
- Bart
--
Bart Smaalders Solaris Kernel Performance
barts at cyber dot eng dot sun dot com http://blogs.sun.com/barts
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
53
From:
Registered:
3/9/05
|
|
|
|
Re: Re: Re: Re: Google Summer of Code proposal - check
it
Posted:
Apr 24, 2006 4:41 PM
in response to: barts
|
|
Bart Smaalders wrote On 04/24/06 14:00,:
> Nikolay Molchanov wrote:
>
>>>Perhaps you misunderstand Electric Cloud.
>>>It provides benefit not through a better make (though
>>>maybe that's part of it), but by filesystem-level dependency
>>>analysis, distribution,and file caching.
>>
>>Oh, yes, I think you are right. BTW, it is interesting to estimate
>>what improvement we can get from file caching. Sun Studio 11
>>compilers have a new feature: "preprocessed include files". This
>>is some kind of caching, and we can try to use it. Another way is
>>to have /usr/include in memory (f.e. copy this directory to /tmp
>>and mount as /usr/include). What else makes sense to cache?
>>
>>Thanks,
>>Nik
>
>
>
> Oblig. caveat: ON doesn't include files from /usr/include
> because it is the source of those include files ;-).
That's ok, so the directory to be cached is the "usr/include"
directory from the build area, correct?
Thanks,
Nik
__________________ Nikolay Molchanov
SUNSTUDIO/DMAKE/MAKE/SCCS, Sun Microsystems
email: nikolay dot molchanov at sun dot com
phone: (650) 786-7643 (x87643)
------------------------------------------------------------
NOTICE: This email message is for the sole use of the
intended recipient(s) and may contain confidential and
privileged information. Any unauthorized review, use,
disclosure or distribution is prohibited.
If you are not the intended recipient, please contact
the sender by reply email and destroy all copies of the
original message.
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,172
From:
US
Registered:
3/9/05
|
|
|
|
Re: Re: Re: Re: Google Summer of Code proposal -
check it
Posted:
Apr 24, 2006 7:53 PM
in response to: nikm
|
|
Nikolay Molchanov wrote:
>
> Bart Smaalders wrote On 04/24/06 14:00,:
>> Nikolay Molchanov wrote:
>>
>>>> Perhaps you misunderstand Electric Cloud.
>>>> It provides benefit not through a better make (though
>>>> maybe that's part of it), but by filesystem-level dependency
>>>> analysis, distribution,and file caching.
>>> Oh, yes, I think you are right. BTW, it is interesting to estimate
>>> what improvement we can get from file caching. Sun Studio 11
>>> compilers have a new feature: "preprocessed include files". This
>>> is some kind of caching, and we can try to use it. Another way is
>>> to have /usr/include in memory (f.e. copy this directory to /tmp
>>> and mount as /usr/include). What else makes sense to cache?
>>>
>>> Thanks,
>>> Nik
>>
>>
>> Oblig. caveat: ON doesn't include files from /usr/include
>> because it is the source of those include files ;-).
>
> That's ok, so the directory to be cached is the "usr/include"
> directory from the build area, correct?
Whatever is passed w/ the -YI flag...
- Bart
--
Bart Smaalders Solaris Kernel Performance
barts at cyber dot eng dot sun dot com http://blogs.sun.com/barts
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,253
From:
Registered:
3/9/05
|
|
|
|
Re: Google Summer of Code proposal - check it out...
Posted:
Apr 21, 2006 2:45 PM
in response to: bustos
|
|
On Fri, Apr 21, 2006 at 01:37:09PM -0700, David Bustos wrote:
> Quoth Mike Pogue on Thu, Apr 20, 2006 at 03:53:43PM -0700:
> > Tools community folks,
> >
> > Over at the Google Summer of Code proposal that OpenSolaris is
> > doing, I'm proposing a "how to get more build parallelism on
> > highly-parallel chips" project (or study or whatever you want to
> > call it) . I've added a short blurb on this over at the wiki,
> > located here:
> >
> > http://www.genunix.org/wiki/index.php/Summer_of_Code#Exploiting_Parallelism_to_Optimize_Build_Performance
>
> Have you heard of Electric Cloud?
Looks pretty cool, though it only supports GNU and Microsoft make
emulation.
It would also be really nice to make the ON build griddable. Dmake
supports distribution via the Sun Grid Engine, and it would be really cool
to be able to take advantage of that. This would be useful even without
finding more parallelism in the build.
Danek
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,174
From:
IL
Registered:
4/27/05
|
|
|
|
Re: Google Summer of Code proposal - check it out...
Posted:
Apr 21, 2006 3:13 PM
in response to: dduvall
|
|
On 4/22/06, Danek Duvall <danek dot duvall at sun dot com> wrote:
> On Fri, Apr 21, 2006 at 01:37:09PM -0700, David Bustos wrote:
>
> > Quoth Mike Pogue on Thu, Apr 20, 2006 at 03:53:43PM -0700:
> > > Tools community folks,
> > >
> > > Over at the Google Summer of Code proposal that OpenSolaris is
> > > doing, I'm proposing a "how to get more build parallelism on
> > > highly-parallel chips" project (or study or whatever you want to
> > > call it) . I've added a short blurb on this over at the wiki,
> > > located here:
> > >
> > > http://www.genunix.org/wiki/index.php/Summer_of_Code#Exploiting_Parallelism_to_Optimize_Build_Performance
> >
> > Have you heard of Electric Cloud?
>
> Looks pretty cool, though it only supports GNU and Microsoft make
> emulation.
>
> It would also be really nice to make the ON build griddable. Dmake
> supports distribution via the Sun Grid Engine, and it would be really cool
> to be able to take advantage of that. This would be useful even without
> finding more parallelism in the build.
I recently came across SCons (http://www.scons.org/). Although it may
be too revolutionary ... Anyway, worth having a look...
(And written in Python ! - while we are on Python subject due to mercurial :-D)
--
Regards,
Cyril
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
16
From:
Registered:
4/8/06
|
|
|
|
Re: Google Summer of Code proposal - check it out...
Posted:
Apr 21, 2006 3:50 PM
in response to: bustos
|
|
On Fri, 2006-04-21 at 13:37 -0700, David Bustos wrote:
> Have you heard of Electric Cloud?
You could also take a look at distcc and ccache. They have the
advantage of being free, already working with gcc, and not caring about
your make environment at all.
<b
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
