|
Replies:
73
-
Last Post:
Oct 11, 2007 1:38 PM
by: kupfer
|
Threads:
[
Previous
|
Next
]
|
|
Alexander Kolba...
akolb@eng.sun.com
|
|
|
|
Proposal: project for build time improvements
Posted:
Mar 23, 2007 5:52 PM
|
|
I would like to propose a project aimed to improve ON build times. This can
live either under tools or under performance or under some other community.
There was some discussion about it a while ago in the context of Google summer
of code:
http://www.opensolaris.org/jive/message.jspa?messageID=33951
- Alex Kolbasov
__
http://blogs.sun.com/akolb
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
Posts:
1,156
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 23, 2007 6:18 PM
in response to: Alexander Kolba...
|
|
Alexander Kolbasov wrote:
> I would like to propose a project aimed to improve ON build times. This can
> live either under tools or under performance or under some other community.
>
> There was some discussion about it a while ago in the context of Google summer
> of code:
>
> http://www.opensolaris.org/jive/message.jspa?messageID=33951
>
>
> - Alex Kolbasov
> __
> http://blogs.sun.com/akolb
>
> _______________________________________________
> tools-discuss mailing list
> tools-discuss at opensolaris dot org
One thought I had when talking about this with Sasha was that this could
perhaps reside as part of the ONNV project. We could setup separate
mailing lists, and a separate repository for it.
I'm fine with it being standalone or as part of the ONNV project - but
Danek & Mark own the ONNV project, so I don't profess to speak for them :)
cheers,
steve
--
stephen lau // stevel at sun dot com | 650.786.0845 | http://whacked.net
opensolaris // solaris kernel development
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
812
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 25, 2007 1:47 PM
in response to: stevel
|
|
On Fri 23 Mar 2007 at 06:18PM, Stephen Lau wrote:
> Alexander Kolbasov wrote:
> >I would like to propose a project aimed to improve ON build times. This
> >can live either under tools or under performance or under some other
> >community.
> >
> >There was some discussion about it a while ago in the context of Google
> >summer of code:
> >
> >http://www.opensolaris.org/jive/message.jspa?messageID=33951
> >
> >
> >- Alex Kolbasov
> >__
> >http://blogs.sun.com/akolb
> >
> >_______________________________________________
> >tools-discuss mailing list
> >tools-discuss at opensolaris dot org
>
> One thought I had when talking about this with Sasha was that this could
> perhaps reside as part of the ONNV project. We could setup separate
> mailing lists, and a separate repository for it.
>
> I'm fine with it being standalone or as part of the ONNV project - but
> Danek & Mark own the ONNV project, so I don't profess to speak for them :)
This could also be a candidate project for the proposed-but-never-
implemented "gardeners" community if there is renewed interest in it. I
proposed it but ran out of time to make it happen.
-dp
--
Daniel Price - Solaris Kernel Engineering - dp at eng dot sun dot com - blogs.sun.com/dp
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
770
From:
US
Registered:
6/17/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 23, 2007 6:21 PM
in response to: Alexander Kolba...
|
|
Alexander Kolbasov wrote:
> I would like to propose a project aimed to improve ON build times. This can
> live either under tools or under performance or under some other community.
>
> There was some discussion about it a while ago in the context of Google summer
> of code:
>
> http://www.opensolaris.org/jive/message.jspa?messageID=33951
>
>
According to Eric[1], you need to Cc project proposals to
opensolaris-discuss. While I don't at all like that way of doing things,
it's probably best if you follow up and Cc there.
[1]
http://mail.opensolaris.org/pipermail/website-discuss/2007-March/003175.html
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,695
From:
US
Registered:
4/28/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 23, 2007 10:28 PM
in response to: richlowe
|
|
On Fri, 23 Mar 2007, Richard Lowe wrote:
>
> ... you need to Cc project proposals to
> opensolaris-discuss. While I don't at all like that way of doing things...
Yes, another hopefully top item for the new OGB's docket.
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
3,046
From:
US
Registered:
3/9/05
|
|
|
|
re: Proposal: project for build time improvements
Posted:
Mar 25, 2007 5:53 AM
in response to: Alexander Kolba...
|
|
> I would like to propose a project aimed to improve ON build times. This
> can live either under tools or under performance or under some other
> community.
>
> There was some discussion about it a while ago in the context of Google
> summer of code:
> http://www.opensolaris.org/jive/message.jspa?messageID=33951
While I certainly support the initiative, I'd like to hear a bit more
about what's planned. For instance, the earlier thread talked mainly
about parallelizing the build, but in my experience, a lot of the time is
wasted because of bad algorithms inside the tools themselves rather than
lack of parallelism. For instance, 6357412 showed how an undersized hash
table inside lint added about 30 minutes to a 90 minute build.
--
meem
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,218
From:
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 25, 2007 8:57 AM
in response to: meem
|
|
Peter Memishian wrote:
> While I certainly support the initiative, I'd like to hear a bit more
> about what's planned. For instance, the earlier thread talked mainly
> about parallelizing the build, but in my experience, a lot of the time is
> wasted because of bad algorithms inside the tools themselves rather than
> lack of parallelism. For instance, 6357412 showed how an undersized hash
> table inside lint added about 30 minutes to a 90 minute build.
I seem to remember the CTF tools were right hogs at one point as well -
I don't know if that is still the case. The main problem with the build
wasn't that it had too little parallelism - in fact in many parts of the
build it had *way* too much, and in other parts it had way too little.
This lead to a distinctly 'lumpy' load profile on the build. What's
probably needed is some sort of queueing mechanism so we can ensure that
the load on a build machine is more constant. As far as I can tell,
this isn't something that our current make offers.
--
Alan Burlison
--
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
3,046
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 25, 2007 9:55 AM
in response to: alanbur
|
|
> What's probably needed is some sort of queueing mechanism so we can
> ensure that the load on a build machine is more constant. As far as I
> can tell, this isn't something that our current make offers.
A while back, Bill Sommerfeld filed:
4694000 recursive parallel make should conspire to limit max #jobs
... which led to dmake being enhanced to monitor load average and throttle
if needed. However, the token-passing scheme used by gmake would probably
be better (as Bill mentioned in that bug report).
--
meem
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,710
From:
NZ
Registered:
4/27/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 25, 2007 1:01 PM
in response to: alanbur
|
|
Alan Burlison wrote:
> Peter Memishian wrote:
>
>> While I certainly support the initiative, I'd like to hear a bit more
>> about what's planned. For instance, the earlier thread talked mainly
>> about parallelizing the build, but in my experience, a lot of the
>> time is
>> wasted because of bad algorithms inside the tools themselves rather than
>> lack of parallelism. For instance, 6357412 showed how an undersized
>> hash
>> table inside lint added about 30 minutes to a 90 minute build.
>
>
> I seem to remember the CTF tools were right hogs at one point as well
> - I don't know if that is still the case. The main problem with the
> build wasn't that it had too little parallelism - in fact in many
> parts of the build it had *way* too much, and in other parts it had
> way too little. This lead to a distinctly 'lumpy' load profile on the
> build. What's probably needed is some sort of queueing mechanism so
> we can ensure that the load on a build machine is more constant. As
> far as I can tell, this isn't something that our current make offers.
>
One aspect of ON builds that continues to bug me is the lack of support
for distributed builds. I have often wondered, but never had the time
to investigate, whether the makefiles could be "flattened out" to enable
independent directories to be built in parallel rather than
sequentialy. In my experience (projects with several hundred C and C++
files in several dozen directories) a single makefile gives dmake a much
better workload.
Ian
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
3,046
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 25, 2007 11:32 PM
in response to: ian
|
|
> One aspect of ON builds that continues to bug me is the lack of support
> for distributed builds. I have often wondered, but never had the time
> to investigate, whether the makefiles could be "flattened out" to enable
> independent directories to be built in parallel rather than
> sequentialy.
I'm confused. Right now, most of the directories under usr/src/lib and
usr/src/cmd are built in parallel as per the .PARALLEL directives in
usr/src/lib/Makefile and usr/src/cmd/Makefile.
--
meem
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,710
From:
NZ
Registered:
4/27/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 26, 2007 12:23 AM
in response to: meem
|
|
Peter Memishian wrote:
> > One aspect of ON builds that continues to bug me is the lack of support
> > for distributed builds. I have often wondered, but never had the time
> > to investigate, whether the makefiles could be "flattened out" to enable
> > independent directories to be built in parallel rather than
> > sequentialy.
>
>I'm confused. Right now, most of the directories under usr/src/lib and
>usr/src/cmd are built in parallel as per the .PARALLEL directives in
>usr/src/lib/Makefile and usr/src/cmd/Makefile.
>
>
>
I'll have to recheck, but I'm sure last time I tried this didn't work as
well as building every object from a single makefile. I'll have to check
how well this works with distributed building as well.
Ian.
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Alexander Kolba...
akolb@eng.sun.com
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 26, 2007 8:36 PM
in response to: meem
|
|
>
> > One aspect of ON builds that continues to bug me is the lack of support
> > for distributed builds. I have often wondered, but never had the time
> > to investigate, whether the makefiles could be "flattened out" to enable
> > independent directories to be built in parallel rather than
> > sequentialy.
>
> I'm confused. Right now, most of the directories under usr/src/lib and
> usr/src/cmd are built in parallel as per the .PARALLEL directives in
> usr/src/lib/Makefile and usr/src/cmd/Makefile.
For example, DEBUG/NON-DEBUG/lint are serial. Also, it seems that uts is done
before lib/cmd. There are, probably, many more interesting opportunities here.
- akolb
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
3,046
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 26, 2007 8:57 PM
in response to: Alexander Kolba...
|
|
> For example, DEBUG/NON-DEBUG/lint are serial.
I'm not sure I follow. Do you mean serialized with respect to each other?
usr/src/Makefile.lint is certainly not serial.
--
meem
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Alexander Kolba...
akolb@eng.sun.com
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 27, 2007 4:55 PM
in response to: meem
|
|
>
> > For example, DEBUG/NON-DEBUG/lint are serial.
>
> I'm not sure I follow. Do you mean serialized with respect to each other?
> usr/src/Makefile.lint is certainly not serial.
Correct, serialized with respect to each other.
- akolb
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,710
From:
NZ
Registered:
4/27/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 27, 2007 1:39 AM
in response to: meem
|
|
Peter Memishian wrote:
> > One aspect of ON builds that continues to bug me is the lack of support
> > for distributed builds. I have often wondered, but never had the time
> > to investigate, whether the makefiles could be "flattened out" to enable
> > independent directories to be built in parallel rather than
> > sequentialy.
>
>I'm confused. Right now, most of the directories under usr/src/lib and
>usr/src/cmd are built in parallel as per the .PARALLEL directives in
>usr/src/lib/Makefile and usr/src/cmd/Makefile.
>
>
>
I just tried running nightly on a dual core AMD 64 box with the
.PARALLEL directives removed form
usr/src/lib/Makefile and usr/src/cmd/Makefile. On this system, the difference was small:
With .PARALLEL:
time nightly ./opensolaris.sh
real 212m14.023s
user 325m5.196s
sys 56m58.105s
Without:
time nightly ./opensolaris.sh
real 228m14.275s
user 325m5.731s
sys 56m55.456s
The picture may be different on a bigger machine.
Ian
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
770
From:
US
Registered:
6/17/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 27, 2007 2:50 AM
in response to: ian
|
|
Ian Collins wrote:
> Peter Memishian wrote:
>
>>> One aspect of ON builds that continues to bug me is the lack of support
>>> for distributed builds. I have often wondered, but never had the time
>>> to investigate, whether the makefiles could be "flattened out" to enable
>>> independent directories to be built in parallel rather than
>>> sequentialy.
>> I'm confused. Right now, most of the directories under usr/src/lib and
>> usr/src/cmd are built in parallel as per the .PARALLEL directives in
>> usr/src/lib/Makefile and usr/src/cmd/Makefile.
>>
>>
>>
> I just tried running nightly on a dual core AMD 64 box with the
> .PARALLEL directives removed form
>
> usr/src/lib/Makefile and usr/src/cmd/Makefile. On this system, the difference was small:
>
> With .PARALLEL:
>
> time nightly ./opensolaris.sh
> real 212m14.023s
> user 325m5.196s
> sys 56m58.105s
>
> Without:
>
> time nightly ./opensolaris.sh
> real 228m14.275s
> user 325m5.731s
> sys 56m55.456s
>
> The picture may be different on a bigger machine.
It will be largely the same, you're at least in part seeing the results of:
4631488 "lib/Makefile is too patient: .WAITs should be reduced"
-- Rich
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,156
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 27, 2007 7:02 AM
in response to: richlowe
|
|
On Tue, Mar 27, 2007 at 05:50:32AM -0400, Richard Lowe wrote:
> Ian Collins wrote:
> >Peter Memishian wrote:
> >
> >>>One aspect of ON builds that continues to bug me is the lack of support
> >>>for distributed builds. I have often wondered, but never had the time
> >>>to investigate, whether the makefiles could be "flattened out" to enable
> >>>independent directories to be built in parallel rather than
> >>>sequentialy.
> >>I'm confused. Right now, most of the directories under usr/src/lib and
> >>usr/src/cmd are built in parallel as per the .PARALLEL directives in
> >>usr/src/lib/Makefile and usr/src/cmd/Makefile.
> >>
> >>
> >>
> >I just tried running nightly on a dual core AMD 64 box with the
> >.PARALLEL directives removed form
> >
> >usr/src/lib/Makefile and usr/src/cmd/Makefile. On this system, the
> >difference was small:
> >
> >With .PARALLEL:
> >
> >time nightly ./opensolaris.sh
> >real 212m14.023s
> >user 325m5.196s
> >sys 56m58.105s
> >
> >Without:
> >
> >time nightly ./opensolaris.sh
> >real 228m14.275s
> >user 325m5.731s
> >sys 56m55.456s
> >
> >The picture may be different on a bigger machine.
>
> It will be largely the same, you're at least in part seeing the results of:
> 4631488 "lib/Makefile is too patient: .WAITs should be reduced"
Jonathan Adams had made some progress on that bug and left behind a
workspace which I pointed Sasha & Jonathan to last week - so I believe
they're looking into addressing that one as part of this build-time
projet.
cheers,
steve
--
stephen lau // stevel at sun dot com | 650.786.0845 | http://whacked.net
opensolaris // solaris kernel development
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,710
From:
NZ
Registered:
4/27/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 26, 2007 2:01 PM
in response to: alanbur
|
|
Alan Burlison wrote:
> Peter Memishian wrote:
>
>> While I certainly support the initiative, I'd like to hear a bit more
>> about what's planned. For instance, the earlier thread talked mainly
>> about parallelizing the build, but in my experience, a lot of the
>> time is
>> wasted because of bad algorithms inside the tools themselves rather than
>> lack of parallelism. For instance, 6357412 showed how an undersized
>> hash
>> table inside lint added about 30 minutes to a 90 minute build.
>
>
> I seem to remember the CTF tools were right hogs at one point as well
> - I don't know if that is still the case. The main problem with the
> build wasn't that it had too little parallelism - in fact in many
> parts of the build it had *way* too much, and in other parts it had
> way too little. This lead to a distinctly 'lumpy' load profile on the
> build. What's probably needed is some sort of queueing mechanism so
> we can ensure that the load on a build machine is more constant. As
> far as I can tell, this isn't something that our current make offers.
>
I just run a vanilla nightly on a dual core AMD 64 box and logged the
number of instances of dmake running during the build. The average for
the first half of the build was 18 (the logger crapped out at this point
with 'fork: Not enough space') and the system load average was about 8.
In my opinion, that is way too many for the machine to build
efficiently. To get the best out of this box, one dmake instance with 4
jobs is the ideal, once it goes beyond 4, the build times increase. So
it does look like there is *way* too much parallelism.
Ian
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
352
From:
AU
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 27, 2007 3:11 AM
in response to: ian
|
|
On 03/26/07 22:01, Ian Collins wrote:
> I just run a vanilla nightly on a dual core AMD 64 box and logged the
> number of instances of dmake running during the build. The average for
> the first half of the build was 18 (the logger crapped out at this point
> with 'fork: Not enough space') and the system load average was about 8.
> In my opinion, that is way too many for the machine to build
> efficiently. To get the best out of this box, one dmake instance with 4
> jobs is the ideal, once it goes beyond 4, the build times increase. So
> it does look like there is *way* too much parallelism.
For many years now I have made the parallelism be 4-8 times the number
of cores present in the system, with a max of 128. You can thrash the
cores almost as much as you like and it does not really hurt build
performance - only ever gains. However if you have limited memory
you need to be a bit more conservative.
So on my sparc build system (20 cores and 48GB) I have max jobs set to 96 -
have done for years and (provided you have local storage) it will build
in under 2 hours (and sparc has a lot of 'unix' directories to build for
each sun4u platform). On x86 systems I use max=24 on my twin opteron
desktop, max=32 on a 4 Opteron v40z, max=64 on a 16 core X4600 M2.
On shared build systems I admin I use FSS such that each user is in their
own project and has equal shares; so, aside from the memory footprint,
someone using max=96 does not spoil the party for other users.
I also use that on my desktop so I can have a max=24 build running
in one project while I continue with interactive use in another -
works a treat.
So provided you have the memory I've generally found it never hurts to
throw parallelism at the ON build. Of course for so much of the
build we serialize or have narrow parallelism, anyway! But where
we get wide - such as the kernel module builds - load averages in
the hundreds do no harm.
Gavin
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,710
From:
NZ
Registered:
4/27/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 27, 2007 4:42 AM
in response to: gavinm
|
|
Gavin Maltby wrote:
> On 03/26/07 22:01, Ian Collins wrote:
>
>> I just run a vanilla nightly on a dual core AMD 64 box and logged the
>> number of instances of dmake running during the build. The average for
>> the first half of the build was 18 (the logger crapped out at this point
>> with 'fork: Not enough space') and the system load average was about
>> 8. In my opinion, that is way too many for the machine to build
>> efficiently. To get the best out of this box, one dmake instance with 4
>> jobs is the ideal, once it goes beyond 4, the build times increase. So
>> it does look like there is *way* too much parallelism.
>
>
> For many years now I have made the parallelism be 4-8 times the number
> of cores present in the system, with a max of 128. You can thrash the
> cores almost as much as you like and it does not really hurt build
> performance - only ever gains. However if you have limited memory
> you need to be a bit more conservative.
>
I tend to spec a build node at 1GB per core. I have found a lower
number of jobs (2) per core tends to result in better job distribution
when doing distributed builds. Beyond that point, the law of
diminishing returns kicks in.
> So on my sparc build system (20 cores and 48GB) I have max jobs set to
> 96 -
> have done for years and (provided you have local storage) it will build
> in under 2 hours (and sparc has a lot of 'unix' directories to build for
> each sun4u platform). On x86 systems I use max=24 on my twin opteron
> desktop, max=32 on a 4 Opteron v40z, max=64 on a 16 core X4600 M2.
>
On my systems, build times flatten out at about 2 per core, so I never
go any higher.
>
> So provided you have the memory I've generally found it never hurts to
> throw parallelism at the ON build. Of course for so much of the
> build we serialize or have narrow parallelism, anyway! But where
> we get wide - such as the kernel module builds - load averages in
> the hundreds do no harm.
>
I'd still like to see ON builds work in a distributed environment, I
tend to use more dual core nodes rather than one big box, it gives
better bang for the buck.
Also some ability to tune the build to match the capabilities of the
build machine would be worth investigating.
Ian
> Gavin
>
>
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
804
From:
Plano, TX
Registered:
4/27/05
|
|
|
|
re: Proposal: project for build time improvements
Posted:
Mar 26, 2007 4:15 AM
in response to: meem
|
|
On Sun, 25 Mar 2007, Peter Memishian wrote:
>
> > I would like to propose a project aimed to improve ON build times. This
> > can live either under tools or under performance or under some other
> > community.
> >
> > There was some discussion about it a while ago in the context of Google
> > summer of code:
> > http://www.opensolaris.org/jive/message.jspa?messageID=33951
>
> While I certainly support the initiative, I'd like to hear a bit more
> about what's planned. For instance, the earlier thread talked mainly
> about parallelizing the build, but in my experience, a lot of the time is
> wasted because of bad algorithms inside the tools themselves rather than
> lack of parallelism. For instance, 6357412 showed how an undersized hash
^^^^^^^
Sun internal only access?
> table inside lint added about 30 minutes to a 90 minute build.
>
>
Al Hopper Logical Approach Inc, Plano, TX. al@logical-approach.com
Voice: 972.379.2133 Fax: 972.379.2134 Timezone: US CDT
OpenSolaris.Org Community Advisory Board (CAB) Member - Apr 2005
OpenSolaris Governing Board (OGB) Member - Feb 2006
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
3,046
From:
US
Registered:
3/9/05
|
|
|
|
re: Proposal: project for build time improvements
Posted:
Mar 26, 2007 4:37 AM
in response to: alhopper
|
|
> > While I certainly support the initiative, I'd like to hear a bit more
> > about what's planned. For instance, the earlier thread talked mainly
> > about parallelizing the build, but in my experience, a lot of the time is
> > wasted because of bad algorithms inside the tools themselves rather than
> > lack of parallelism. For instance, 6357412 showed how an undersized hash
> ^^^^^^^
> Sun internal only access?
IIRC, bugs with only internal call records are not visible externally for
some reason. (Or maybe there's another explanation?)
--
meem
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
5,506
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 26, 2007 8:07 AM
in response to: alhopper
|
|
Al Hopper wrote:
> On Sun, 25 Mar 2007, Peter Memishian wrote:
>
>> > I would like to propose a project aimed to improve ON build times. This
>> > can live either under tools or under performance or under some other
>> > community.
>> >
>> > There was some discussion about it a while ago in the context of Google
>> > summer of code:
>> > http://www.opensolaris.org/jive/message.jspa?messageID=33951
>>
>> While I certainly support the initiative, I'd like to hear a bit more
>> about what's planned. For instance, the earlier thread talked mainly
>> about parallelizing the build, but in my experience, a lot of the time is
>> wasted because of bad algorithms inside the tools themselves rather than
>> lack of parallelism. For instance, 6357412 showed how an undersized hash
> ^^^^^^^
> Sun internal only access?
It's a lint bug - lint is part of the Sun Studio compiler suite, which is
not open.
--
-Alan Coopersmith- alan dot coopersmith at sun dot com
Sun Microsystems, Inc. - X Window System Engineering
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
283
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 26, 2007 10:17 AM
in response to: alanc
|
|
FYI:
Sun Studio bugs are private by default. You can add a "jdcinclude"
to make them public, but it's a manual process. Some Sun Studio
bugs refer to competitive information about SPEC benchmarks, or upcoming
hardware and we can't open those ones.
--chris
Alan Coopersmith wrote:
> Al Hopper wrote:
>> On Sun, 25 Mar 2007, Peter Memishian wrote:
>>
>>> > I would like to propose a project aimed to improve ON build times.
>>> This
>>> > can live either under tools or under performance or under some other
>>> > community.
>>> >
>>> > There was some discussion about it a while ago in the context of
>>> Google
>>> > summer of code:
>>> > http://www.opensolaris.org/jive/message.jspa?messageID=33951
>>>
>>> While I certainly support the initiative, I'd like to hear a bit more
>>> about what's planned. For instance, the earlier thread talked mainly
>>> about parallelizing the build, but in my experience, a lot of the
>>> time is
>>> wasted because of bad algorithms inside the tools themselves rather than
>>> lack of parallelism. For instance, 6357412 showed how an undersized
>>> hash
>> ^^^^^^^
>> Sun internal only access?
>
> It's a lint bug - lint is part of the Sun Studio compiler suite, which is
> not open.
>
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
56
From:
Registered:
7/14/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 26, 2007 10:25 AM
in response to: quenelle
|
|
Note that it takes like 24 hours for a bug to show up on bugs.sun.com
On Mon, 26 Mar 2007, Chris Quenelle wrote:
>
> FYI:
>
> Sun Studio bugs are private by default. You can add a "jdcinclude"
> to make them public, but it's a manual process. Some Sun Studio
> bugs refer to competitive information about SPEC benchmarks, or upcoming
> hardware and we can't open those ones.
>
> --chris
>
>
> Alan Coopersmith wrote:
>> Al Hopper wrote:
>>> On Sun, 25 Mar 2007, Peter Memishian wrote:
>>>
>>>> > I would like to propose a project aimed to improve ON build times.
>>>> This
>>>> > can live either under tools or under performance or under some
>>>> other
>>>> > community.
>>>> >
>>>> > There was some discussion about it a while ago in the context of
>>>> Google
>>>> > summer of code:
>>>> > http://www.opensolaris.org/jive/message.jspa?messageID=33951
>>>>
>>>> While I certainly support the initiative, I'd like to hear a bit more
>>>> about what's planned. For instance, the earlier thread talked mainly
>>>> about parallelizing the build, but in my experience, a lot of the time
>>>> is
>>>> wasted because of bad algorithms inside the tools themselves rather
>>>> than
>>>> lack of parallelism. For instance, 6357412 showed how an undersized
>>>> hash
>>> ^^^^^^^
>>> Sun internal only access?
>>
>> It's a lint bug - lint is part of the Sun Studio compiler suite, which is
>> not open.
>>
> _______________________________________________
> tools-discuss mailing list
> tools-discuss at opensolaris dot org
>
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 26, 2007 11:10 AM
in response to: meem
|
|
* Peter Memishian <peter dot memishian at sun dot com> [2007-03-25 05:54]:
>
> > I would like to propose a project aimed to improve ON build times. This
> > can live either under tools or under performance or under some other
> > community.
> >
> > There was some discussion about it a while ago in the context of Google
> > summer of code:
> > http://www.opensolaris.org/jive/message.jspa?messageID=33951
>
> While I certainly support the initiative, I'd like to hear a bit more
> about what's planned.
Me, too. When people ask me about the build, I always suggest (a)
investigating overlapping the kernel build/lint and the
library/userland build phases, (b) investigating applicability of
ccache and distcc, and (c) assessing a kernel Makefile rewrite--all
after reading Alan's paper on measuring the build. It would be nice
to hear if any of these are being considered, or what might be pursued
in their stead. A project that only seeks to increase naive
parallelism won't much help developers on modest systems.
Also, there's always the possibility of eliminating bulk of the DEBUG
build by looking to DTrace as a means to implement kernel assertions.
- Stephen
--
Stephen Hahn, PhD Solaris Kernel Development, Sun Microsystems
stephen dot hahn at sun dot com http://blogs.sun.com/sch/
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
15
From:
Broomfield CO
Registered:
7/24/06
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 29, 2007 12:23 PM
in response to: Stephen Hahn
|
|
I hadn't been paying as much attention to tools-discuss lately, but this
is an area I am *very* interested in.
What I have done so far is prototype a modular Makefile build system.
This doesn't ever recurse into directories, it in effect builds one
massive directed graph of what is to be built. I had only got as far as
install_h and building several libs (including libc).
The results for that were very encouraging. I could "dmake -j 10000"
with success. Of course that actually slowed the build down since 10000
jobs with only 4 cores is a tad excessive.
I'm kind of swamped with gate stuff right now. Here is how things kind
of look right now.
So here is the way the directories are laid out (use fixed width font to
veiw):
ws/
|
usr/-------+-------build/
| |
closed/---+--src/ +-----+-----+
| | |
Mk/ gnu/ sunw/
|
i386/---+---sparc/
|
+---------+--------+-------+--------+
| | | | |
Makefile arch.mk gen/ debug/ release/
|
+---------+----------+---------+
| | | |
proto/ archives/ packages/ usr/
The source tree is untouched (usr/src, usr/closed).
You can build both debug and non-debug bits at the same time with the
same workspace.
Additioinally, you can use build gnu/studio at the same time with the
same workspace.
And finally, you can use the same workspace over NFS to build both i386
and sparc ISAs at the same time.
So if you are building debug bits with Sun compiler on i386, object
files will be placed in build/sunw/i386/debug/usr/*
I only got a few libs going before getting distracted however. I also
only focused on x86/studio.
But it did simultaneously build debug/release. And it did spin up way
more jobs. It is very parallel.
The plan is to get genunix building and then get Mike Sullivan/meem and
other build masters of ON to check it out. Although I did demo building
the libs for Danek.
Some good background reading:
http://aegis.sourceforge.net/auug97.pdf
I used Peter Miller's idea and had initially set up a gmake system you
can read about here:
http://homepage.mac.com/david.marker/makefile.html
Goes without saying the ON system uses dmake and is much cooler ;)
-dvd
Stephen Hahn wrote:
> * Peter Memishian <peter dot memishian at sun dot com> [2007-03-25 05:54]:
>
>> > I would like to propose a project aimed to improve ON build times. This
>> > can live either under tools or under performance or under some other
>> > community.
>> >
>> > There was some discussion about it a while ago in the context of Google
>> > summer of code:
>> > http://www.opensolaris.org/jive/message.jspa?messageID=33951
>>
>> While I certainly support the initiative, I'd like to hear a bit more
>> about what's planned.
>>
>
> Me, too. When people ask me about the build, I always suggest (a)
> investigating overlapping the kernel build/lint and the
> library/userland build phases, (b) investigating applicability of
> ccache and distcc, and (c) assessing a kernel Makefile rewrite--all
> after reading Alan's paper on measuring the build. It would be nice
> to hear if any of these are being considered, or what might be pursued
> in their stead. A project that only seeks to increase naive
> parallelism won't much help developers on modest systems.
>
> Also, there's always the possibility of eliminating bulk of the DEBUG
> build by looking to DTrace as a means to implement kernel assertions.
>
> - Stephen
>
>
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,710
From:
NZ
Registered:
4/27/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 29, 2007 3:03 PM
in response to: dm120769
|
|
David Marker wrote:
> I hadn't been paying as much attention to tools-discuss lately, but
> this is an area I am *very* interested in.
>
> What I have done so far is prototype a modular Makefile build system.
> This doesn't ever recurse into directories, it in effect builds one
> massive directed graph of what is to be built. I had only got as far
> as install_h and building several libs (including libc).
>
How did this work? Did you end up with one monolithic makefile? If so,
it should be a ideal candidate for distributed building.
Ian
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
15
From:
Broomfield CO
Registered:
7/24/06
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 29, 2007 3:09 PM
in response to: ian
|
|
No, it isn't one big monolithic Makefile.
It is one Makefile that includes many others. So it is still ideal for
distributed building, since dmake can build one huge directed graph.
After the 1st prototype I realized I did some things the hard way and
some things just plain wrong. So I'm re-tooling in my spare time and
should have something I'm not as embarrassed to let others take a peek
at "soon" (whatever that means when you are gatekeeping :)
-dvd
Ian Collins wrote:
> David Marker wrote:
>
>
>> I hadn't been paying as much attention to tools-discuss lately, but
>> this is an area I am *very* interested in.
>>
>> What I have done so far is prototype a modular Makefile build system.
>> This doesn't ever recurse into directories, it in effect builds one
>> massive directed graph of what is to be built. I had only got as far
>> as install_h and building several libs (including libc).
>>
>>
> How did this work? Did you end up with one monolithic makefile? If so,
> it should be a ideal candidate for distributed building.
>
> Ian
>
>
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,710
From:
NZ
Registered:
4/27/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 29, 2007 3:20 PM
in response to: dm120769
|
|
David Marker wrote:
> No, it isn't one big monolithic Makefile.
> It is one Makefile that includes many others. So it is still ideal for
> distributed building, since dmake can build one huge directed graph.
> After the 1st prototype I realized I did some things the hard way and
> some things just plain wrong. So I'm re-tooling in my spare time and
> should have something I'm not as embarrassed to let others take a peek
> at "soon" (whatever that means when you are gatekeeping :)
> -dvd
>
So you end up with a single instance of dmake running the entire build,
rather than the scores of competing instances spawned by the current
build system?
I look forward to seeing it.
Ian
> Ian Collins wrote:
>
>> David Marker wrote:
>>
>>
>>
>>> I hadn't been paying as much attention to tools-discuss lately, but
>>> this is an area I am *very* interested in.
>>>
>>> What I have done so far is prototype a modular Makefile build system.
>>> This doesn't ever recurse into directories, it in effect builds one
>>> massive directed graph of what is to be built. I had only got as far
>>> as install_h and building several libs (including libc).
>>>
>>>
>>
>> How did this work? Did you end up with one monolithic makefile? If so,
>> it should be a ideal candidate for distributed building.
>>
>> Ian
>>
>>
>
>
>
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
15
From:
Broomfield CO
Registered:
7/24/06
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 29, 2007 3:38 PM
in response to: ian
|
|
Ian Collins wrote:
> So you end up with a single instance of dmake running the entire build,
> rather than the scores of competing instances spawned by the current
> build system?
>
Correct. Only one instance of dmake.
-dvd
> I look forward to seeing it.
>
> Ian
>
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Alexander Kolba...
akolb@eng.sun.com
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 30, 2007 10:47 AM
in response to: dm120769
|
|
> I hadn't been paying as much attention to tools-discuss lately, but this
> is an area I am *very* interested in.
>
> What I have done so far is prototype a modular Makefile build system.
> This doesn't ever recurse into directories, it in effect builds one
> massive directed graph of what is to be built. I had only got as far as
> install_h and building several libs (including libc).
>
...
> Some good background reading:
> http://aegis.sourceforge.net/auug97.pdf
>
> I used Peter Miller's idea and had initially set up a gmake system you
> can read about here:
> http://homepage.mac.com/david.marker/makefile.html
This is quite interesting! It seems that this approach requires wholesale
rewrite of the whole Make system and even the relatively straightforward lex
example in the last link was rather complicated and depended on some fine
nuances of GNU make. Do you think that some form of incremental approach is
possible where some targets may be converted to the flat Makefiles and some
remain as is, so that the conversion may happen gradually?
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
824
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 30, 2007 11:10 AM
in response to: Alexander Kolba...
|
|
>>>>> "akolb" == Alexander Kolbasov <akolb at eng dot sun dot com> writes:
akolb> This is quite interesting!
Indeed. I would love to get away from recursive makes--they've caused
me quite a bit of grief over the past couple years.
akolb> Do you think that some form of incremental approach is possible
akolb> where some targets may be converted to the flat Makefiles and
akolb> some remain as is, so that the conversion may happen gradually?
When I read the Miller paper some months ago, I thought a little bit
about how this could be done for ON. But I haven't done any of the
actual implementation work that Dave has.
I think it would work to start near the leaf nodes of the build tree and
work up. For example, if usr/src/lib/libfoo has been converted,
usr/src/lib/Makefile can still do a recursive make until everything
under usr/src/lib has been converted.
mike
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,710
From:
NZ
Registered:
4/27/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 30, 2007 12:29 PM
in response to: Alexander Kolba...
|
|
Alexander Kolbasov wrote:
>
>This is quite interesting! It seems that this approach requires wholesale
>rewrite of the whole Make system and even the relatively straightforward lex
>example in the last link was rather complicated and depended on some fine
>nuances of GNU make. Do you think that some form of incremental approach is
>possible where some targets may be converted to the flat Makefiles and some
>remain as is, so that the conversion may happen gradually?
>
>
>
Isn't the use of gmake a retrograde step? Surely we should be working
to optimise the use of dmake to exploit distributed building?
Ian
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
15
From:
Broomfield CO
Registered:
7/24/06
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 30, 2007 3:11 PM
in response to: ian
|
|
The link (http://homepage.mac.com/david.marker/makefile.html) I pointed
you at was work I had done a long long time ago, in a galaxy far far away.
The current work does use dmake, Danek (rightly) smacked down the use of
gmake from day 1. And in my opinion the work has benefited _massively_
from that decision. When I used gmake I was very temped to use all sorts
of GNU-make extensions, which it turns out you don't need in order to
create a modular non-recursive build system. And to be honest it would
take me hours to re-decipher the huge gmake functions in that link (you
can't add comments in a gmake function).
The new method uses very small shell scripts (_not_ embedded in the
makefile where they are impossible to read and understand), with the
goal that they should all be smaller than the CDDL wherever possible.
Its hard to explain how it all fits together without an example or some
pictures, so I should probably bug Stephen Lau for help on setting
something up on OpenSolaris.org. So far this has been my "Midnight
Engineering", aka "After School Project" while I do my best on the
gatekeeping staff, but as there is so much interest perhaps it is time
to get what I have available. I would prefer for version 2 to be
finished 1st (I learned a lot just by doing the initial demo).
-dvd
Ian Collins wrote:
> Alexander Kolbasov wrote:
>
>
>> This is quite interesting! It seems that this approach requires wholesale
>> rewrite of the whole Make system and even the relatively straightforward lex
>> example in the last link was rather complicated and depended on some fine
>> nuances of GNU make. Do you think that some form of incremental approach is
>> possible where some targets may be converted to the flat Makefiles and some
>> remain as is, so that the conversion may happen gradually?
>>
>>
>>
>>
> Isn't the use of gmake a retrograde step? Surely we should be working
> to optimise the use of dmake to exploit distributed building?
>
> Ian
>
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,710
From:
NZ
Registered:
4/27/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 30, 2007 3:35 PM
in response to: dm120769
|
|
David Marker wrote:
> The link (http://homepage.mac.com/david.marker/makefile.html) I
> pointed you at was work I had done a long long time ago, in a galaxy
> far far away.
>
> The current work does use dmake, Danek (rightly) smacked down the use
> of gmake from day 1. And in my opinion the work has benefited
> _massively_ from that decision. When I used gmake I was very temped to
> use all sorts of GNU-make extensions, which it turns out you don't
> need in order to create a modular non-recursive build system. And to
> be honest it would take me hours to re-decipher the huge gmake
> functions in that link (you can't add comments in a gmake function).
>
> The new method uses very small shell scripts (_not_ embedded in the
> makefile where they are impossible to read and understand), with the
> goal that they should all be smaller than the CDDL wherever possible.
> Its hard to explain how it all fits together without an example or
> some pictures, so I should probably bug Stephen Lau for help on
> setting something up on OpenSolaris.org. So far this has been my
> "Midnight Engineering", aka "After School Project" while I do my best
> on the gatekeeping staff, but as there is so much interest perhaps it
> is time to get what I have available. I would prefer for version 2 to
> be finished 1st (I learned a lot just by doing the initial demo).
>
That's all good news!
The first thing I have done on just about every project I've worked on
over the past (bloody hell!) 20 years is re-engineer the build system,
so I'm very interested in seing someone else's work.
Cheers,
Ian
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
494
From:
US
Registered:
6/14/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Sep 9, 2007 6:17 PM
in response to: dm120769
To: Communities » tools » discuss
|
|
Is anyone still working on this? Last message seems to be from March. I'd like to assist if there's still ongoing work, or if not, just take a crack at it (just don't want to duplicate efforts).
|
|
|
|
|
Alexander Kolba...
akolb@eng.sun.com
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Sep 12, 2007 11:18 AM
in response to: jbk
|
|
> Is anyone still working on this? Last message seems to be from March. I'd like to assist if there's still ongoing work, or if not, just take a crack at it (just don't want to duplicate efforts).
Hi Jason,
Yes, this is still an active project. We have a little page for it at
http://www.opensolaris.org/os/project/onnv/onnv_build/faster_builds
Currently we are trying to increase build parallelism by compiling more parts
in parallel and we we need to figure out various hidden build dependencies
that create build-time race conditions.
We also have nice DTrace based tools that allow you to get extensive data from
your build and graph it.
It would be great if you can participate in this work. What are your primary
interests? What are you thinking off?
Regards,
- akolb
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
976
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Sep 12, 2007 12:49 PM
in response to: Alexander Kolba...
|
|
On Wed, 2007-09-12 at 11:18 -0700, Alexander Kolbasov wrote:
> Yes, this is still an active project.
Is there a project mailing list we can join? I don't see a link to it
on the project page.
- Bill
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Alexander Kolba...
akolb@eng.sun.com
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Sep 12, 2007 4:09 PM
in response to: sommerfe
|
|
> On Wed, 2007-09-12 at 11:18 -0700, Alexander Kolbasov wrote:
> > Yes, this is still an active project.
>
> Is there a project mailing list we can join? I don't see a link to it
> on the project page.
There is no mailing list - tools-discuss was supposed to be the one. We'll
keep all discussions related to the project at tools-discuss.
- akolb
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
494
From:
US
Registered:
6/14/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Sep 12, 2007 1:10 PM
in response to: Alexander Kolba...
|
|
On 9/12/07, Alexander Kolbasov <akolb at eng dot sun dot com> wrote:
> > Is anyone still working on this? Last message seems to be from March. I'd like to assist if there's still ongoing work, or if not, just take a crack at it (just don't want to duplicate efforts).
>
> Hi Jason,
>
> Yes, this is still an active project. We have a little page for it at
>
> http://www.opensolaris.org/os/project/onnv/onnv_build/faster_builds
>
> Currently we are trying to increase build parallelism by compiling more parts
> in parallel and we we need to figure out various hidden build dependencies
> that create build-time race conditions.
>
> We also have nice DTrace based tools that allow you to get extensive data from
> your build and graph it.
>
> It would be great if you can participate in this work. What are your primary
> interests? What are you thinking off?
>
> Regards,
>
> - akolb
I really liked the idea (obviously this would probably have to be done
in a gradual fashion) of the recursive-make-is-bad approach -- i.e.
generate the complete dependency tree and allow dmake to go wild with
it. That should I think expose all opportunities for parallelism.
Somewhat related, having attempted an ON build, as well as doing fbsd
& linux builds in the past, I'm wondering if there might be some
opportunities to simplify the process as well? I'm not familiar
enough with all the different use cases to really make any sort of
informed opinion, so I'd rather see what others think.
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Alexander Kolba...
akolb@eng.sun.com
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Sep 12, 2007 4:11 PM
in response to: jbk
|
|
> > http://www.opensolaris.org/os/project/onnv/onnv_build/faster_builds
> >
> > Currently we are trying to increase build parallelism by compiling more parts
> > in parallel and we we need to figure out various hidden build dependencies
> > that create build-time race conditions.
>
> I really liked the idea (obviously this would probably have to be done
> in a gradual fashion) of the recursive-make-is-bad approach -- i.e.
> generate the complete dependency tree and allow dmake to go wild with
> it. That should I think expose all opportunities for parallelism.
It is a very promising idea and Mark have a nice prototype for lib. It would
be great if you can explore this possibility and see how can this be
integrated with our existing build system.
> Somewhat related, having attempted an ON build, as well as doing fbsd
> & linux builds in the past, I'm wondering if there might be some
> opportunities to simplify the process as well? I'm not familiar
> enough with all the different use cases to really make any sort of
> informed opinion, so I'd rather see what others think.
What parts do you think are worth simplification?
- Sasha
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
494
From:
US
Registered:
6/14/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Sep 12, 2007 5:08 PM
in response to: Alexander Kolba...
|
|
On 9/12/07, Alexander Kolbasov <akolb at eng dot sun dot com> wrote:
> > > http://www.opensolaris.org/os/project/onnv/onnv_build/faster_builds
> > >
> > > Currently we are trying to increase build parallelism by compiling more parts
> > > in parallel and we we need to figure out various hidden build dependencies
> > > that create build-time race conditions.
> >
> > I really liked the idea (obviously this would probably have to be done
> > in a gradual fashion) of the recursive-make-is-bad approach -- i.e.
> > generate the complete dependency tree and allow dmake to go wild with
> > it. That should I think expose all opportunities for parallelism.
>
> It is a very promising idea and Mark have a nice prototype for lib. It would
> be great if you can explore this possibility and see how can this be
> integrated with our existing build system.
I'd love to see what's been done if anyone can put it somewhere to look at.
>
> > Somewhat related, having attempted an ON build, as well as doing fbsd
> > & linux builds in the past, I'm wondering if there might be some
> > opportunities to simplify the process as well? I'm not familiar
> > enough with all the different use cases to really make any sort of
> > informed opinion, so I'd rather see what others think.
>
> What parts do you think are worth simplification?
>
> - Sasha
It seems there's a large number of environment variables that have to
be set to get things to even start. It seems to be a bit of a black
art to figure out which ones are needed and which ones are optional.
I'm just wondering if perhaps some defaults here could help.
However, it could be the current level of complexity is the minimum
needed to support the various use cases -- hence why I thought it
might be worth discussing.
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,253
From:
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Sep 12, 2007 7:35 PM
in response to: Alexander Kolba...
|
|
On Wed, Sep 12, 2007 at 04:11:43PM -0700, Alexander Kolbasov wrote:
> > I really liked the idea (obviously this would probably have to be done
> > in a gradual fashion) of the recursive-make-is-bad approach -- i.e.
> > generate the complete dependency tree and allow dmake to go wild with
> > it. That should I think expose all opportunities for parallelism.
>
> It is a very promising idea and Mark have a nice prototype for lib. It
> would be great if you can explore this possibility and see how can this
> be integrated with our existing build system.
Were you able to incorporate any of the work that Dave Marker did on this?
As I recall, he had libraries building with a non-recursive make system,
and did so quite fast.
Danek
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
494
From:
US
Registered:
6/14/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Sep 12, 2007 7:50 PM
in response to: dduvall
|
|
On 9/12/07, Danek Duvall <danek dot duvall at sun dot com> wrote:
> On Wed, Sep 12, 2007 at 04:11:43PM -0700, Alexander Kolbasov wrote:
>
> > > I really liked the idea (obviously this would probably have to be done
> > > in a gradual fashion) of the recursive-make-is-bad approach -- i.e.
> > > generate the complete dependency tree and allow dmake to go wild with
> > > it. That should I think expose all opportunities for parallelism.
> >
> > It is a very promising idea and Mark have a nice prototype for lib. It
> > would be great if you can explore this possibility and see how can this
> > be integrated with our existing build system.
>
> Were you able to incorporate any of the work that Dave Marker did on this?
> As I recall, he had libraries building with a non-recursive make system,
> and did so quite fast.
>
> Danek
>
I know I have not, though I'd be very interested to see what been done.
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
3,046
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Sep 13, 2007 1:05 AM
in response to: dduvall
|
|
> Were you able to incorporate any of the work that Dave Marker did on this?
> As I recall, he had libraries building with a non-recursive make system,
> and did so quite fast.
I haven't seen Dave Marker's work on this -- nor am I sure this is
related -- but I mentioned to Sasha (Alexender Kolbasov) another
idea that Jonathan, Rod Evans and I were kicking around, as covered
in Jonathan's Evaluation for:
4631488 lib/Makefile is too patient: .WAITs should be reduced
Make the library build two-phase. During the first phase, "dummy"
libraries would be created from the information in the spec files, and
installed in some central location. In the second phase, everything
would be built in parallel, but would link against the "dummy"
libraries.
It looks like we've got almost everything we need already, except for
exported data. We would have to add a 'definition' line for each such
element. Then:
1. use spec2map -F to generate the "filter" version of the mapfile,
mapfile.dummy
2. process all of the 'data' entries in the spec file, and generate
a .c file containing all of the data definitions.
3. Compile the .c file, then link it with -M mapfile.dummy, to generate
libfoo.so.1.dummy
4. install libfoo.so.1.dummy as '$(DUMMYDIR)/libfoo.so.1', and add the
'libfoo.so' symlink.
(The reason you need the data definitions and the .c lines is to get
the data sizes correct)
Note that this Evaluation was written back when we had specfiles, but I'm
sure we could do something similar (and perhaps simpler) with mapfiles.
Again, I inherited 4631488 from Jonathan but haven't had time to work on
it. If others are ready and able, I'm happy to hand it off.
--
meem
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Alexander Kolba...
akolb@eng.sun.com
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Sep 21, 2007 4:54 PM
in response to: dduvall
|
|
Trying to improve parallelism we are building uts/i86pc and uts/intel in
parallel. Since both rely on intel/genunix we build it upfront. When doing so
we noticed that later when various intel drivers do ctfmerge with genunix
sometimes ctfmerges fail because the genunix file is corrupt. It seems that
for some reason genunix file is built more than once. Does anyone have some
ideas on why this is happening? Is there any way to get dmake to tell why it
decided to rebuild genunix?
- akolb
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,174
From:
IL
Registered:
4/27/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Sep 22, 2007 12:29 AM
in response to: Alexander Kolba...
|
|
On 9/22/07, Alexander Kolbasov <akolb at eng dot sun dot com> wrote:
> Trying to improve parallelism we are building uts/i86pc and uts/intel in
> parallel. Since both rely on intel/genunix we build it upfront. When doing so
> we noticed that later when various intel drivers do ctfmerge with genunix
> sometimes ctfmerges fail because the genunix file is corrupt. It seems that
> for some reason genunix file is built more than once. Does anyone have some
> ideas on why this is happening? Is there any way to get dmake to tell why it
> decided to rebuild genunix?
may these help ?
-d Displays the reasons why make chooses to
rebuild a target. make displays any and all
dependencies that are newer. In addition,
make displays options read in from the
MAKEFLAGS environment variable.
-dd Displays the dependency check and processing
in vast detail.
--
Regards,
Cyril
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Alexander Kolba...
akolb@eng.sun.com
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Oct 4, 2007 11:59 AM
in response to: dduvall
|
|
Here are the changes we suggest doing so far.
They cover the following bugs/RFEs:
6591892 ipf/netinet/Makefile incorrectly tries to install ip_icmp.h
6591900 Various sun4u platforms try to install sys symlink in usr/share/src/uts in parallel
6592974 Kernel can compile in parallel with libraries
6592975 Sparc platforms can be compiled in parallel
6592976 Intel platforms can be compiled in parallel
6592977 sun4u sub-platforms can be compiled in parallel
The webrev is at http://cr.grommit.com/~akolb/build/webrev/index.html
it is based on build 74.
- akolb
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Oct 4, 2007 12:20 PM
in response to: Alexander Kolba...
|
|
* Alexander Kolbasov <akolb at eng dot sun dot com> [2007-10-04 18:59]:
> Here are the changes we suggest doing so far.
>
> They cover the following bugs/RFEs:
>
> 6591892 ipf/netinet/Makefile incorrectly tries to install ip_icmp.h
> 6591900 Various sun4u platforms try to install sys symlink in usr/share/src/uts in parallel
> 6592974 Kernel can compile in parallel with libraries
> 6592975 Sparc platforms can be compiled in parallel
> 6592976 Intel platforms can be compiled in parallel
> 6592977 sun4u sub-platforms can be compiled in parallel
>
> The webrev is at http://cr.grommit.com/~akolb/build/webrev/index.html
>
> it is based on build 74.
Measured improvement is what?
- Stephen
--
sch at sun dot com http://blogs.sun.com/sch/
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
42
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Oct 4, 2007 2:00 PM
in response to: Stephen Hahn
|
|
Stephen Hahn wrote:
> * Alexander Kolbasov <akolb at eng dot sun dot com> [2007-10-04 18:59]:
>
>> Here are the changes we suggest doing so far.
>>
>> They cover the following bugs/RFEs:
>>
>> 6591892 ipf/netinet/Makefile incorrectly tries to install ip_icmp.h
>> 6591900 Various sun4u platforms try to install sys symlink in usr/share/src/uts in parallel
>> 6592974 Kernel can compile in parallel with libraries
>> 6592975 Sparc platforms can be compiled in parallel
>> 6592976 Intel platforms can be compiled in parallel
>> 6592977 sun4u sub-platforms can be compiled in parallel
>>
>> The webrev is at http://cr.grommit.com/~akolb/build/webrev/index.html
>>
>> it is based on build 74.
>>
>
> Measured improvement is what?
>
On a V890 with 16 1.2Ghz UltraSPARC IV cores, the changes reduce a full
nightly(1) build (debug, non-debug, lint, and everything) by ~33% from
3:42 to 2:30 and increased the parallelism of the build from ~6.34 to ~9.88.
Sasha has the numbers for x86/amd64. The improvement there is smaller
because the build on x86/amd64 is much simpler and more parallel than
SPARC. The other way to put this is that the SPARC build has *lots* of
room for improvement which may be an understatement. ;-)
The improvements are mostly from building uts, lib, sun4v, sun4u
(including all of the sun4u platforms), and sparc in parallel on SPARC
and uts, lib, intel, and i86pc in parallel on x86/amd64. This was
accomplished by enabling the SPARC and Intel to build anything that
needs to be built before building everything (else) in parallel for uts,
removing unnecessary .WAIT's, and fixing wherever necessary.
There is definitely more improvment possible too. For example, the
parallelism for usr/src/lib is very poor. That's why we overlapped
building it with uts for now, but it needs fixing. Some other low
hanging fruit is trying to build debug and non-debug in parallel which
might be fun. Anyway, we have focused on making one build work well
first since that will help debug or non-debug.
Finally, the current challenge is verifying that there are no races
somehow. Of course, we are starting another round of stress testing,
increasing the max dmake jobs, doing more builds sequentially and in
parallel, etc.. However, we are wondering whether there is a better way
to determine that there are no races such as determining the dependency
graph and whether the make files express the dependencies correctly.
Implicit dependencies really throw a monkey wrench in trying to make the
build more parallel.... :-(
Jonathan
PS
We need to update our web page with the latest data, scripts, etc., so
we can just point there next time.
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Oct 4, 2007 4:11 PM
in response to: jjc
|
|
* Jonathan Chew <Jonathan dot Chew at sun dot com> [2007-10-04 21:00]:
> Stephen Hahn wrote:
> >* Alexander Kolbasov <akolb at eng dot sun dot com> [2007-10-04 18:59]:
> >
> >>Here are the changes we suggest doing so far.
> >>
> >>They cover the following bugs/RFEs:
> >>
> >>6591892 ipf/netinet/Makefile incorrectly tries to install ip_icmp.h
> >>6591900 Various sun4u platforms try to install sys symlink in
> >>usr/share/src/uts in parallel
> >>6592974 Kernel can compile in parallel with libraries
> >>6592975 Sparc platforms can be compiled in parallel
> >>6592976 Intel platforms can be compiled in parallel
> >>6592977 sun4u sub-platforms can be compiled in parallel
> >>
> >>The webrev is at http://cr.grommit.com/~akolb/build/webrev/index.html
> >>
> >>it is based on build 74.
> >
> > Measured improvement is what?
>
> On a V890 with 16 1.2Ghz UltraSPARC IV cores, the changes reduce a full
> nightly(1) build (debug, non-debug, lint, and everything) by ~33% from
> 3:42 to 2:30 and increased the parallelism of the build from ~6.34 to ~9.88.
Great, thanks.
- Stephen
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
6,813
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Oct 4, 2007 1:38 PM
in response to: Alexander Kolba...
|
|
Alexander Kolbasov writes:
> The webrev is at http://cr.grommit.com/~akolb/build/webrev/index.html
I looked at all of the files, so if I didn't mention one, it means I
have no comments.
usr/src/Makefile
57,62: what's the distinction between PARALLEL_DIRS and
PARALLEL_DIRS_EXTRA? If someone were to add a new directory here,
how would he choose one over the other?
152,153: nit: I might have used "FRC" here.
usr/src/uts/intel/Makefile
101: why don't we end up building genunix twice?
usr/src/uts/sun4u/Makefile
120,122: I don't think "FRC" does anything here, does it? The
target directory should already have a dependency on FRC so that we
will descend during a normal build.
--
James Carlson, Solaris Networking <james dot d dot carlson at sun dot com>
Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Alexander Kolba...
akolb@eng.sun.com
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Oct 5, 2007 2:47 PM
in response to: carlsonj
|
|
James,
thank you for your comments!
> Alexander Kolbasov writes:
> > The webrev is at http://cr.grommit.com/~akolb/build/webrev/index.html
>
> I looked at all of the files, so if I didn't mention one, it means I
> have no comments.
>
> usr/src/Makefile
>
> 57,62: what's the distinction between PARALLEL_DIRS and
> PARALLEL_DIRS_EXTRA? If someone were to add a new directory here,
> how would he choose one over the other?
This is an interesting piece. We wanted to achieve two things:
1) Move bits that are built in parallel now into a macro
So we simply rewrite the parallelism of header construction with
a macro expansion.
2) Define a new macro that allows uts/lib parallelism (and may be later it can
include some other parallelism).
3) The two parallel lists above should be separate since there is no
parallelism between the two sets.
This allows users to selectively manipulate these lists using environment
variables - e.g. it is easy to build uts/lib sequentially by overwriting
PARALLEL_DIRS_EXTRA in the environment.
Adding a new directory depends on the parallelism you want to achieve - if it
is extra parallelism to building headers you go with PARALLEL_DIRS and if it
is extra parallelism with uts/lib than you go with PARALLEL_DIRS_EXTRA.
Probably the name choise is not very descriptive. We can change PARALLEL_DIRS
to something like PARALLEL_HEADERS and I am not sure what may be a better name
for uts/lib parallelism macro.
> 152,153: nit: I might have used "FRC" here.
What does FRC achieve in .PARALLEL directives?
> usr/src/uts/intel/Makefile
>
> 101: why don't we end up building genunix twice?
We end up going to genunix directory twice (actually, three times - one extra
time when we build i86pc) but we don't build anything there because it is
already up to date.
> usr/src/uts/sun4u/Makefile
>
> 120,122: I don't think "FRC" does anything here, does it? The
> target directory should already have a dependency on FRC so that we
> will descend during a normal build.
I think you are right here - FRC doesn't do anything useful here.
- akolb
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
6,813
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Oct 5, 2007 3:12 PM
in response to: Alexander Kolba...
|
|
Alexander Kolbasov writes:
> > Alexander Kolbasov writes:
> > > The webrev is at http://cr.grommit.com/~akolb/build/webrev/index.html
[...]
> > I looked at all of the files, so if I didn't mention one, it means I
> > have no comments.
> >
> > usr/src/Makefile
> >
> > 57,62: what's the distinction between PARALLEL_DIRS and
> > PARALLEL_DIRS_EXTRA? If someone were to add a new directory here,
> > how would he choose one over the other?
[...]
> 3) The two parallel lists above should be separate since there is no
> parallelism between the two sets.
Ah, ok. That's the clincher for me.
> > 152,153: nit: I might have used "FRC" here.
>
> What does FRC achieve in .PARALLEL directives?
I'm suggesting using "FRC" instead of "DUMMY" as the filler target to
avoid doing everything in parallel if the macro is empty.
> > usr/src/uts/intel/Makefile
> >
> > 101: why don't we end up building genunix twice?
>
> We end up going to genunix directory twice (actually, three times - one extra
> time when we build i86pc) but we don't build anything there because it is
> already up to date.
OK. Just checking to make sure I understood.
> > usr/src/uts/sun4u/Makefile
> >
> > 120,122: I don't think "FRC" does anything here, does it? The
> > target directory should already have a dependency on FRC so that we
> > will descend during a normal build.
>
> I think you are right here - FRC doesn't do anything useful here.
OK.
--
James Carlson, Solaris Networking <james dot d dot carlson at sun dot com>
Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
James C. McPher...
James.McPherson@Sun....
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Oct 4, 2007 4:29 PM
in response to: Alexander Kolba...
|
|
Alexander Kolbasov wrote:
> Here are the changes we suggest doing so far.
>
> They cover the following bugs/RFEs:
>
> 6591892 ipf/netinet/Makefile incorrectly tries to install ip_icmp.h
> 6591900 Various sun4u platforms try to install sys symlink in usr/share/src/uts in parallel
> 6592974 Kernel can compile in parallel with libraries
> 6592975 Sparc platforms can be compiled in parallel
> 6592976 Intel platforms can be compiled in parallel
> 6592977 sun4u sub-platforms can be compiled in parallel
>
> The webrev is at http://cr.grommit.com/~akolb/build/webrev/index.html
> it is based on build 74.
Hi Sasha,
one niggle and one question:
- the webrev script that you used appears to have empty
frames versions of the diffs
- do you have a plan to backport these changes to an
S10 Update release? Should this be considered at all?
thankyou,
James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Alexander Kolba...
akolb@eng.sun.com
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Oct 4, 2007 5:40 PM
in response to: James C. McPher...
|
|
James,
> > The webrev is at http://cr.grommit.com/~akolb/build/webrev/index.html
> > it is based on build 74.
>
> Hi Sasha,
> one niggle and one question:
>
> - the webrev script that you used appears to have empty
> frames versions of the diffs
Hmm,
the frames are there when I check them locally, but for some reason they do
not show up on cr.grommit.com - I have no idea why.
> - do you have a plan to backport these changes to an
> S10 Update release? Should this be considered at all?
Hmm,
it could, definitely, be a possibility, although it is a bit strange since it
should not result in any patch produced, so the change is strictly internal.
This seems to be rather unusual candidate for a back-port.
- akolb
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
James C. McPher...
James.McPherson@Sun....
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Oct 4, 2007 5:41 PM
in response to: Alexander Kolba...
|
|
Alexander Kolbasov wrote:
> James,
...
>> - do you have a plan to backport these changes to an
>> S10 Update release? Should this be considered at all?
> it could, definitely, be a possibility, although it is a bit strange since it
> should not result in any patch produced, so the change is strictly internal.
> This seems to be rather unusual candidate for a back-port.
That is true, but I reckon people doing backports
to S10Update and RPE would all like to see much
faster build times. I know I would, that's for sure!
cheers,
James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
352
From:
AU
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Oct 5, 2007 12:55 AM
in response to: James C. McPher...
|
|
|
|
On 10/05/07 01:41, James C. McPherson wrote:
> Alexander Kolbasov wrote:
>> James,
> ...
>>> - do you have a plan to backport these changes to an
>>> S10 Update release? Should this be considered at all?
>> it could, definitely, be a possibility, although it is a bit strange since it
>> should not result in any patch produced, so the change is strictly internal.
>> This seems to be rather unusual candidate for a back-port.
>
>
> That is true, but I reckon people doing backports
> to S10Update and RPE would all like to see much
> faster build times. I know I would, that's for sure!
I don't believe you'll find the sustaining org, who owm the S10
gates, will want any overhaul of the S10 Makefile structure
*even if* it halves buildtimes. There is now a standardized
build environment etc for S10 in an effort to stabilize
the build process, make it reproducable and so on and
juggling Makefiles is extremely difficult to quantify
in terms of risk (even if you can somehow prove the
changes correct you're still subject to make bugs,
speaking of which I think S10 still uses ParallelMake
instead of dmake).
So for the sake of our customers, just throw hardware at
S10 builds for sustaining/RPE. It's much cheaper than
debugging some weird bugs that a Makefile reorg may bring.
Gavin
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
James C. McPher...
James.McPherson@Sun....
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Oct 5, 2007 4:08 AM
in response to: gavinm
|
|
Gavin Maltby wrote:
> On 10/05/07 01:41, James C. McPherson wrote:
>> Alexander Kolbasov wrote:
>>> James,
>> ...
>>>> - do you have a plan to backport these changes to an
>>>> S10 Update release? Should this be considered at all?
>>> it could, definitely, be a possibility, although it is a bit strange
>>> since it should not result in any patch produced, so the change is
>>> strictly internal. This seems to be rather unusual candidate for a
>>> back-port.
>>
>>
>> That is true, but I reckon people doing backports
>> to S10Update and RPE would all like to see much
>> faster build times. I know I would, that's for sure!
>
> I don't believe you'll find the sustaining org, who owm the S10
> gates, will want any overhaul of the S10 Makefile structure
> *even if* it halves buildtimes. There is now a standardized
> build environment etc for S10 in an effort to stabilize
> the build process, make it reproducable and so on and
> juggling Makefiles is extremely difficult to quantify
> in terms of risk (even if you can somehow prove the
> changes correct you're still subject to make bugs,
> speaking of which I think S10 still uses ParallelMake
> instead of dmake).
>
> So for the sake of our customers, just throw hardware at
> S10 builds for sustaining/RPE. It's much cheaper than
> debugging some weird bugs that a Makefile reorg may bring.
As much as I'd like to see the changes go
into an S10 update, I am very, very conscious
that these have a system-wide effect and
thus would probably not get a look-in. While
the S10 build system is slow, it isn't broken.
Although... perhaps if the changes demonstrated
a 90%+ improvement in build times..... :-)
cheers,
James
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
39
From:
GB
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Oct 5, 2007 8:13 AM
in response to: James C. McPher...
|
|
As an RPE engineer I'd rather the S10 gates weren't messed with too
much. Gavin is right that we try to minimise change where possible.
There should be a customer benefit for us to want to change the patch gates.
Chris
James C. McPherson wrote:
>
> As much as I'd like to see the changes go
> into an S10 update, I am very, very conscious
> that these have a system-wide effect and
> thus would probably not get a look-in. While
> the S10 build system is slow, it isn't broken.
>
>
> Although... perhaps if the changes demonstrated
> a 90%+ improvement in build times..... :-)
>
>
> cheers,
> James
> --
> Senior Kernel Software Engineer, Solaris
> Sun Microsystems
> _______________________________________________
> tools-discuss mailing list
> tools-discuss at opensolaris dot org
>
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
824
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Oct 11, 2007 1:38 PM
in response to: Alexander Kolba...
|
|
>>>>> "akolb" == Alexander Kolbasov <akolb at eng dot sun dot com> writes:
>> - the webrev script that you used appears to have empty
>> frames versions of the diffs
akolb> the frames are there when I check them locally, but for some
akolb> reason they do not show up on cr.grommit.com - I have no idea
akolb> why.
We had a similar problem with cr.opensolaris.org. IIRC, Steve Lau
tracked down to some bad interactions between the site stylesheet and
webrev. I believe Steve fixed it for cr.opensolaris.org, so you might
want to post there instead of on grommit.
mike
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
James C. McPher...
James.McPherson@Sun....
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Oct 4, 2007 4:33 PM
in response to: Alexander Kolba...
|
|
Alexander Kolbasov wrote:
> Here are the changes we suggest doing so far.
>
> They cover the following bugs/RFEs:
>
> 6591892 ipf/netinet/Makefile incorrectly tries to install ip_icmp.h
> 6591900 Various sun4u platforms try to install sys symlink in usr/share/src/uts in parallel
> 6592974 Kernel can compile in parallel with libraries
> 6592975 Sparc platforms can be compiled in parallel
> 6592976 Intel platforms can be compiled in parallel
> 6592977 sun4u sub-platforms can be compiled in parallel
Another question - what change is seen using these
Makefile mods on single-cpu systems?
James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
976
From:
US
Registered:
3/9/05
|
|
|
|
More kernel makefile performance tuning.
Posted:
Oct 6, 2007 7:19 AM
in response to: Alexander Kolba...
|
|
Some months ago I mentioned the world of hurt which comes out of the uts
makefiles using lots of wildcards to find source files.
Last night I got inspired, did some prototyping, and I now have real
data on how much benefit we get from making the wildcards go away.
While what I have is not even close to complete, the prototype has
already successfully built all sparc kernel modules (admittedly, I was
holding it together with the moral equivalent of duct tape and
clothespins. It flew to pieces when I attempted a nightly, but I
expected that). But the performance improvement is so dramatic I really
had to share this now..
One key microbenchmark relevant to developer productivity is the time it
takes to do an incremental build that doesn't actually build anything.
Currently, on a 8x2x1.35ghz V890, a null build of uts/sun4u/genunix
takes:
dmake 30.06s user 2.54s system 88% cpu 36.815 total
With my prototype (which constructs a genunix which links
successfully...) I have it down to:
dmake 2.65s user 0.41s system 81% cpu 3.735 total
More importantly, a complete no-op build in usr/src/uts on the
aforementioned V890 takes just under 3 minutes. And that's without
Sasha's changes to increase parallelism.
What's going on? Well, the hudnreds of wildcard rules in the various
Makefile.rules cause dmake to go hunting through hundreds of source
directories for each source file.
If you expand the wildcards once -- which is straightforward to get 95%
right with a perl script because the *.rules files are so stylized --
you eliminate all that hunting around.
As always, there are a few weirdo special cases that need manual
cleanup; it also turns out that a bunch of the wildcard rules are either
duplicates or completely pointless (no source files matched).
I filed
6613884 Wildcards in uts/.../Makefile.rules result in O(n^2) build
behavior
Oh, in case anyone's wondering, it looks like my changes don't touch any
of the makefiles Sasha is changing.
Making this real:
I'm going to take the "mkrules" script, and split it in two.
The first half will take a Makefile.rules file and generate an condensed
intermediate Makefile.rule.in file which will be human-maintainable and
which can be fed to the second script.
The second half will take the *.in file and expand all the wildcards
into a "Makefile.rule" file.
This project would:
add the scripts
add the Makefile.rule.in files
add rules to the top-level uts/Makefile to generate
Makefile.rule files
change everything that includes Makefile.rules to include
Makefile.rule instead
delete the Makefile.rules files.
The reason for the rename to Makefile.rule is to avoid causing trouble
for projects which have unintegrated changes to Makefile.rules files
when this project integrates.
- Bill
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
39
From:
GB
Registered:
3/9/05
|
|
|
|
Re: More kernel makefile performance tuning.
Posted:
Oct 6, 2007 7:49 AM
in response to: sommerfe
|
|
Wow - what a fantastic improvement.
One thing I will caution though is for projects wanting to take code
back to Solaris 10, they'll have to do two different types of Makefile
changes. But really well worth the additional work
Chris
Bill Sommerfeld wrote:
> Some months ago I mentioned the world of hurt which comes out of the uts
> makefiles using lots of wildcards to find source files.
>
> Last night I got inspired, did some prototyping, and I now have real
> data on how much benefit we get from making the wildcards go away.
>
> While what I have is not even close to complete, the prototype has
> already successfully built all sparc kernel modules (admittedly, I was
> holding it together with the moral equivalent of duct tape and
> clothespins. It flew to pieces when I attempted a nightly, but I
> expected that). But the performance improvement is so dramatic I really
> had to share this now..
>
> One key microbenchmark relevant to developer productivity is the time it
> takes to do an incremental build that doesn't actually build anything.
>
> Currently, on a 8x2x1.35ghz V890, a null build of uts/sun4u/genunix
> takes:
>
> dmake 30.06s user 2.54s system 88% cpu 36.815 total
>
> With my prototype (which constructs a genunix which links
> successfully...) I have it down to:
>
> dmake 2.65s user 0.41s system 81% cpu 3.735 total
>
> More importantly, a complete no-op build in usr/src/uts on the
> aforementioned V890 takes just under 3 minutes. And that's without
> Sasha's changes to increase parallelism.
>
> What's going on? Well, the hudnreds of wildcard rules in the various
> Makefile.rules cause dmake to go hunting through hundreds of source
> directories for each source file.
>
> If you expand the wildcards once -- which is straightforward to get 95%
> right with a perl script because the *.rules files are so stylized --
> you eliminate all that hunting around.
>
> As always, there are a few weirdo special cases that need manual
> cleanup; it also turns out that a bunch of the wildcard rules are either
> duplicates or completely pointless (no source files matched).
>
> I filed
>
> 6613884 Wildcards in uts/.../Makefile.rules result in O(n^2) build
> behavior
>
> Oh, in case anyone's wondering, it looks like my changes don't touch any
> of the makefiles Sasha is changing.
>
> Making this real:
>
> I'm going to take the "mkrules" script, and split it in two.
>
> The first half will take a Makefile.rules file and generate an condensed
> intermediate Makefile.rule.in file which will be human-maintainable and
> which can be fed to the second script.
>
> The second half will take the *.in file and expand all the wildcards
> into a "Makefile.rule" file.
>
> This project would:
> add the scripts
> add the Makefile.rule.in files
> add rules to the top-level uts/Makefile to generate
> Makefile.rule files
> change everything that includes Makefile.rules to include
> Makefile.rule instead
> delete the Makefile.rules files.
>
> The reason for the rename to Makefile.rule is to avoid causing trouble
> for projects which have unintegrated changes to Makefile.rules files
> when this project integrates.
>
> - Bill
>
>
>
> _______________________________________________
> tools-discuss mailing list
> tools-discuss at opensolaris dot org
>
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
976
From:
US
Registered:
3/9/05
|
|
|
|
Re: More kernel makefile performance tuning.
Posted:
Oct 6, 2007 8:13 AM
in response to: cwb
|
|
On Sat, 2007-10-06 at 15:49 +0100, Chris Beal wrote:
> One thing I will caution though is for projects wanting to take code
> back to Solaris 10, they'll have to do two different types of Makefile
> changes.
Fortunately, the extra work will be minimal.
This is a purely local optimization to the makefiles with no new
external dependencies once the makefile is expanded.
While not ideal from either an aesthetic or a build performance
standpoint, you can mix the current wildcard rules with explicit rules
for files in other directories. I know this works -- I'm doing that now
because my tools don't recognize and thus haven't eliminated all the
wildcards yet.
I'd imagine that projects introducing new source directories in their
backports to earlier releases would be able to extract the
auto-generated rules for the files in their new source directories,
paste them into new or existing Makefile.rules, and get some of the
performance benefit within their new turf.
Or they could recreate the old-style wildcards.
- Bill
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
3,657
From:
DE
Registered:
6/16/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Sep 12, 2007 4:54 PM
in response to: jbk
|
|
Jason King wrote:
> On 9/12/07, Alexander Kolbasov <akolb at eng dot sun dot com> wrote:
[snip]
> > It would be great if you can participate in this work. What are your primary
> > interests? What are you thinking off?
>
> I really liked the idea (obviously this would probably have to be done
> in a gradual fashion) of the recursive-make-is-bad approach -- i.e.
> generate the complete dependency tree and allow dmake to go wild with
> it. That should I think expose all opportunities for parallelism.
AFAIK this will not work as you would expect... make&&dmake scale badly
if you add more and more make rules&&targets, e.g. the time to parse the
makefile and ".make.state" becomes exponentially worse as seen with
libast's Makefile.com which is very large and needs almost 45sec just
for reading on my old SPARC. If you put more entries into a single
Makefile this will IMO kill all improvements unless you call this
Makefile only a few times during the build.
----
Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland dot mainz at nrubsig dot org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 7950090
(;O/ \/ \O;)
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Alexander Kolba...
akolb@eng.sun.com
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 26, 2007 8:34 PM
in response to: meem
|
|
>
> > I would like to propose a project aimed to improve ON build times. This
> > can live either under tools or under performance or under some other
> > community.
> >
> > There was some discussion about it a while ago in the context of Google
> > summer of code:
> > http://www.opensolaris.org/jive/message.jspa?messageID=33951
>
> While I certainly support the initiative, I'd like to hear a bit more
> about what's planned. For instance, the earlier thread talked mainly
> about parallelizing the build, but in my experience, a lot of the time is
> wasted because of bad algorithms inside the tools themselves rather than
> lack of parallelism. For instance, 6357412 showed how an undersized hash
> table inside lint added about 30 minutes to a 90 minute build.
The plan is to take a fresh look at the problem and to identify and fix low
hanging fruits. Exploring parallelism is high on the agenda as well as fixing
Makefiles and exploring some aggressive schemes like caching. I don't think
that we should specifically limit the scope, though.
- akolb
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Alexander Kolba...
akolb@eng.sun.com
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 28, 2007 4:12 PM
in response to: meem
|
|
I did some experiments on 2.6Gz 8-CPu dual-core AMD box.
Here are the results:
* Full build - 2:36:10 real time
o Non-DEBUG build:
+ real 47:15
+ user 2:31:20
+ sys 1:01:47
o DEBUG
+ real 29:57
+ user 1:29:16
+ sys 27:39
o Lint
+ real 1:17:16
+ user 2:31:33
+ sys 9:19
* Incremental build, *no lint* - 0:42:35 real time
o Non-DEBUG
+ real 21:30
+ user 27:46
+ sys 10:39
o DEBUG
+ real 19:28.2
+ user 27:51.4
+ sys 10:32.3
* Clobber non-DEBUG build - 0:57:42
o Non-DEBUG
+ real 49:05
+ user 2:40:06
+ sys 1:01:09
o Non-DEBUG, no shadow gcc - 0:55:39
+ real 47:40
+ user 1:51:37
+ sys 44:31
* Clobber DEBUG build - 0:58:19
o DEBUG
+ real 50:11
+ user 2:45:10
+ sys 1:02:46
* Lint build - 1:03:52
o Lint
+ real 1:03:25
+ user 2:03:55
+ sys 9:22
o S11-new lint - 0:12:26
+ real 11:55
+ user 23:53
+ sys 9:20
This shows that we have ~4.4x parallelism for normal build with shadow gcc and
~2.8 for build without shadow gcc. Lint parallelism is about 2 and the New
lint from S11-new which fixes "6357412 lint takes forever to perform global
kernel crosschecks" makes a huge difference.
--
akolb
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
3,046
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 29, 2007 6:06 AM
in response to: Alexander Kolba...
|
|
> * Lint build - 1:03:52
> o Lint
> + real 1:03:25
> + user 2:03:55
> + sys 9:22
> o S11-new lint - 0:12:26
> + real 11:55
> + user 23:53
> + sys 9:20
>
[ ... ]
> and the New lint from S11-new which fixes "6357412 lint takes forever
> to perform global kernel crosschecks" makes a huge difference.
Indeed -- and I suspect there are other cases of low-hanging fruit. (In
the case of lint, it was literally a matter of resizing a hash table.) So
before we get too obsessed with .PARALLEL and other ways to spread out the
cycles needed by the build, we need to dig into whether those cycles need
to be spent in the first place.
--
meem
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
976
From:
US
Registered:
3/9/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 29, 2007 6:39 AM
in response to: meem
|
|
On Thu, 2007-03-29 at 21:06 +0800, Peter Memishian wrote:
> Indeed -- and I suspect there are other cases of low-hanging fruit. (In
> the case of lint, it was literally a matter of resizing a hash table.) So
> before we get too obsessed with .PARALLEL and other ways to spread out the
> cycles needed by the build, we need to dig into whether those cycles need
> to be spent in the first place.
Indeed.
Another factor is the amount of work done by make itself. One instance
I noticed: If you "truss" an incremental "dmake" of genunix, you will
see an incredible flurry of "stat" calls for source files which don't
exist and will never exist; this is a an artifact of the extensive use
of wildcards in uts/common/Makefile.rules -- there's one wildcard build
rule for each source directory. This means that kernel module makefiles
take time proportional to the square of the total number of source files
to process.
- Bill
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
47
From:
US
Registered:
5/12/05
|
|
|
|
Re: Proposal: project for build time improvements
Posted:
Mar 29, 2007 10:42 AM
in response to: sommerfe
To: Communities » tools » discuss
|
|
> ...
> this is a an artifact of the extensive use
> of wildcards in uts/common/Makefile.rules -- there's one wildcard build
> rule for each source directory. This means that kernel module makefiles
> take time proportional to the square of the total number of source files
> to process.
Very interesting! Bill, would it be possible for you to do a quick before-and-after test run on a single directory (and post the results), so we could see the potential effect of changing this?
Thanks, Mike
|
|
|
|
|
Posts:
976
From:
US
Registered:
3/9/05
|
|
|
|
Re: Re: Proposal: project for build time improvements
Posted:
Mar 29, 2007 12:49 PM
in response to: mpogue
|
|
On Thu, 2007-03-29 at 10:42 -0700, Mike Pogue wrote:
> > ...
> > this is a an artifact of the extensive use
> > of wildcards in uts/common/Makefile.rules -- there's one wildcard build
> > rule for each source directory. This means that kernel module makefiles
> > take time proportional to the square of the total number of source files
> > to process.
>
> Very interesting! Bill, would it be possible for you to do a quick
> before-and-after test run on a single directory (and post the
> results), so we could see the potential effect of changing this?
so, a full fix here would require a substantial rewrite of the kernel
makefiles (not very amenable to a quick before-and-after test!)
But an idea of the level of payoff can be measured by nuking the rules
that aren't needed for a small module.
In the following, I'm measuring how long it takes dmake to realize it
doesn't actually have to do anything in a "hot cache" situation (where
the incremental build was run recently).
On zhadum.east.sun.com, a 1.2ghz V880, a null build in uts/sparc/ipsec
for a zfs-based workspace takes just under 2s:
% time dmake all; time dmake all; time dmake all; time dmake all
dmake all 1.23s user 0.30s system 86% cpu 1.770 total
dmake all 1.25s user 0.30s system 86% cpu 1.784 total
dmake all 1.22s user 0.30s system 86% cpu 1.766 total
dmake all 1.22s user 0.29s system 85% cpu 1.761 total
(I'm running this on a busy build machine; look at the user+system time
numbers and ignore the wall-clock figures)
All of the *.c files built into uts/sparc/ipsec live in
uts/common/inet/ip; if I blow away all of the *.c->*.o rules in
uts/common/Makefile.rules save the one for
$(UTSBASE)/common/inet/ip/%.c, I get:
% time dmake all; time dmake all; time dmake all; time dmake all
dmake all 0.77s user 0.27s system 70% cpu 1.468 total
dmake all 0.78s user 0.26s system 76% cpu 1.357 total
dmake all 0.78s user 0.26s system 73% cpu 1.423 total
dmake all 0.78s user 0.26s system 66% cpu 1.560 total
But it turns out that we're reading in a total of five other
uts/.../Makefile.rules files:
% truss dmake all |& egrep Makefile.rules | egrep open | sort -u
open64("../../common/Makefile.rules", O_RDONLY) = 7
open64("../../sparc/Makefile.rules", O_RDONLY) = 7
open64("../../sparc/v9/Makefile.rules", O_RDONLY) = 7
open64("../../sun/Makefile.rules", O_RDONLY) = 7
open64("../../sun4/Makefile.rules", O_RDONLY) = 7
open64("../../sun4u/Makefile.rules", O_RDONLY) = 7
applying the same treatment to these as well and we shave off some more
time:
% time dmake all; time dmake all; time dmake all; time dmake all
dmake all 0.66s user 0.26s system 34% cpu 2.647 total
dmake all 0.69s user 0.29s system 66% cpu 1.476 total
dmake all 0.66s user 0.24s system 68% cpu 1.322 total
dmake all 0.68s user 0.24s system 47% cpu 1.922 total
so we go from ~1.52s to ~0.92 in cpu time by nuking those wildcard
rules.
More interestingly, we cut the number of mis-aimed file lookups from
1065 to 25:
before:
% truss -f dmake all |& egrep ENOENT | wc -l
1065
After:
% truss -f dmake all |& egrep ENOENT | wc -l
25
The exact impact varies depending on the filesystem - 1.5x speedup with
local zfs, and 1.8x speedup over NFS. I suspect the impact in a "cold
cache" situation will be substantially greater but that would require a
more carefully constructed test.
I think this has a lot of promise...
So, what could we do instead of these wildcards?
Before we used the wildcards, each source file had its own .c -> .o rule
in the various Makefile.rules files; this was a pain to maintain, so the
wildcard scheme was introduced by
4421489 use implicit rules rather than explicit ones
during development of SunOS 5.9
I don't think we want to go back there. But I have a few ideas..
Because kernel module builds are so stylized, the one I think has the
most promise in the long run is to come up with a higher-level way to
describe which source files go into each of the many kernel modules, and
then translate that into actual makefiles. This would resemble some of
the mechanics of the traditional *BSD "config" (but wouldn't be used for
building monolithic kernels, of course!). This would also let us move
to a "recursive make considered harmful"-style build environment which
will allow for better management of build parallelism and further speed
the make..
- Bill
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
Posts:
24
From:
Burlington MA
Registered:
7/7/06
|
|
|
|
Re: Proposal: project for build time improvements
(too many stat calls...)
Posted:
Mar 30, 2007 1:17 PM
in response to: sommerfe
|
|
There may be a relatively simple improvements possible to remedy
the accidental stat'ing due to pattern match rules. Here's one idea:
Use just one (or only a few) pattern match rules, containing a path macro
that's set differently for each module. Actually, a lot of modules
already set
CONF_SRCDIR, so maybe we could just use that with a rule like:
$(OBJS_DIR)/%.o : $(CONF_SRCDIR)/%.c
Maybe it's better to define some new macro names for this, and
add them to per-module makefiles. Also, modules that need
more complicated rules (multiple source locations?) could just
add the other pattern match rules to the module makefile.
Actually, that's another possible remedy to this problem.
Why are the pattern match rules in the common Makefile.rules
if each applies to only one module? Just moving them into the
module-specific makefiles might solve this as well.
Bill Sommerfeld wrote:
> On Thu, 2007-03-29 at 10:42 -0700, Mike Pogue wrote:
>
>>> ...
>>> this is a an artifact of the extensive use
>>> of wildcards in uts/common/Makefile.rules -- there's one wildcard build
>>> rule for each source directory. This means that kernel module makefiles
>>> take time proportional to the square of the total number of source files
>>> to process.
>>>
>> Very interesting! Bill, would it be possible for you to do a quick
>> before-and-after test run on a single directory (and post the
>> results), so we could see the potential effect of changing this?
>>
>
> so, a full fix here would require a substantial rewrite of the kernel
> makefiles (not very amenable to a quick before-and-after test!)
> But an idea of the level of payoff can be measured by nuking the rules
> that aren't needed for a small module.
>
> In the following, I'm measuring how long it takes dmake to realize it
> doesn't actually have to do anything in a "hot cache" situation (where
> the incremental build was run recently).
>
> On zhadum.east.sun.com, a 1.2ghz V880, a null build in uts/sparc/ipsec
> for a zfs-based workspace takes just under 2s:
>
> % time dmake all; time dmake all; time dmake all; time dmake all
> dmake all 1.23s user 0.30s system 86% cpu 1.770 total
> dmake all 1.25s user 0.30s system 86% cpu 1.784 total
> dmake all 1.22s user 0.30s system 86% cpu 1.766 total
> dmake all 1.22s user 0.29s system 85% cpu 1.761 total
>
> (I'm running this on a busy build machine; look at the user+system time
> numbers and ignore the wall-clock figures)
>
> All of the *.c files built into uts/sparc/ipsec live in
> uts/common/inet/ip; if I blow away all of the *.c->*.o rules in
> uts/common/Makefile.rules save the one for
> $(UTSBASE)/common/inet/ip/%.c, I get:
>
> % time dmake all; time dmake all; time dmake all; time dmake all
> dmake all 0.77s user 0.27s system 70% cpu 1.468 total
> dmake all 0.78s user 0.26s system 76% cpu 1.357 total
> dmake all 0.78s user 0.26s system 73% cpu 1.423 total
> dmake all 0.78s user 0.26s system 66% cpu 1.560 total
>
> But it turns out that we're reading in a total of five other
> uts/.../Makefile.rules files:
>
> % truss dmake all |& egrep Makefile.rules | egrep open | sort -u
> open64("../../common/Makefile.rules", O_RDONLY) = 7
> open64("../../sparc/Makefile.rules", O_RDONLY) = 7
> open64("../../sparc/v9/Makefile.rules", O_RDONLY) = 7
> open64("../../sun/Makefile.rules", O_RDONLY) = 7
> open64("../../sun4/Makefile.rules", O_RDONLY) = 7
> open64("../../sun4u/Makefile.rules", O_RDONLY) = 7
>
> applying the same treatment to these as well and we shave off some more
> time:
>
> % time dmake all; time dmake all; time dmake all; time dmake all
> dmake all 0.66s user 0.26s system 34% cpu 2.647 total
> dmake all 0.69s user 0.29s system 66% cpu 1.476 total
> dmake all 0.66s user 0.24s system 68% cpu 1.322 total
> dmake all 0.68s user 0.24s system 47% cpu 1.922 total
>
> so we go from ~1.52s to ~0.92 in cpu time by nuking those wildcard
> rules.
>
> More interestingly, we cut the number of mis-aimed file lookups from
> 1065 to 25:
>
> before:
>
> % truss -f dmake all |& egrep ENOENT | wc -l
> 1065
>
> After:
> % truss -f dmake all |& egrep ENOENT | wc -l
> 25
>
> The exact impact varies depending on the filesystem - 1.5x speedup with
> local zfs, and 1.8x speedup over NFS. I suspect the impact in a "cold
> cache" situation will be substantially greater but that would require a
> more carefully constructed test.
>
> I think this has a lot of promise...
>
> So, what could we do instead of these wildcards?
>
> Before we used the wildcards, each source file had its own .c -> .o rule
> in the various Makefile.rules files; this was a pain to maintain, so the
> wildcard scheme was introduced by
>
> 4421489 use implicit rules rather than explicit ones
>
> during development of SunOS 5.9
>
> I don't think we want to go back there. But I have a few ideas..
>
> Because kernel module builds are so stylized, the one I think has the
> most promise in the long run is to come up with a higher-level way to
> describe which source files go into each of the many kernel modules, and
> then translate that into actual makefiles. This would resemble some of
> the mechanics of the traditional *BSD "config" (but wouldn't be used for
> building monolithic kernels, of course!). This would also let us move
> to a "recursive make considered harmful"-style build environment which
> will allow for better management of build parallelism and further speed
> the make..
>
> - Bill
>
>
> _______________________________________________
> tools-discuss mailing list
> tools-discuss at opensolaris dot org
>
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org
|
|
|
|
|
