@(#)make_std.txt 1.10 94/06/24 SMI # # Copyright 1994 Sun Microsystems, Inc. All rights reserved. # Use is subject to license terms. # Makefile Standards and Guidelines ================================= Introduction ============ The following standards and guidelines were produced in an effort to meet a number of build design goals for SunOS 5.x. The design goals should help you form a framework for understanding and remembering the guidelines. If you do not have the time to study the design goals, then jump down to the guidelines and follow them, using the list of standard macros as a reference or glossary. Many of your questions may be answered by inferring what you can from Glenn Skinner's sample command Makefile at the end of this document. Build Design Goals ================== Incremental install targets: "smart" incremental install targets, sensitive to the modification dates of installed objects and their sources. Incremental builds: "fast" incremental builds by retaining intermediate files, repre- senting intermediate dependencies in targets and rules, and avoiding the clobbering of early build files by later ones. Host architecture independence: a closed system of source and tools designed to avoid corrupting the build machine. support for concurrent work on more than one release on one host. Suitably global flexibility: a hierarchy of centralized "include" makefiles defining standard macros, rules, and targets for their areas of the source tree. great reliance upon central rules and macro names defined in the make.rules file (in the /usr/share/lib/make directory). Configurability: use of standard macro names defined in the shell environment. opportunity to configure local builds by overriding macro values on the command line rather than editing makefiles. AT&T similarity: reasonably close adherence to the Sun and AT&T makefile agreement. Goal Implementation =================== Let us examine some of the implementation details which correspond to each of the goals. A library example will be used. The details in the example have been taken from a wide variety of makefiles just for illus- tration. You should not expect to find nor write all of the workings of the example in one makefile. A command makefile example occurs at the end of these guidelines. Incremental install targets --------------------------- Incremental install targets list the installed objects as dependencies. If an incremental install target is up-to-date and you try rebuilding it, *nothing* happens: dependencies get checked, but no installation occurs. The typical incremental install target depends on the "all" target and a list of target objects in target directories specified in terms of ROOT. Each installed object depends on a local object in the source that can be built using the "all" target. Suppose LIBRARY is "libexample.a" and ROOT is /proto. In this fictional example, if the installed object is to be /proto/usr/lib/libexample.a, the install target would be: install: all $(ROOTLIBRARY) where ROOTLIBRARY is $(ROOTLIBDIR)/$(LIBRARY), fitting this pattern- matching rule: $(ROOTLIBDIR)/%: % $(INS.file) The pattern '%' matches the $(LIBRARY) built by the "all" target and having the value libexample.a. The dependency relationship this defines ensures that the installation will occur *only* if the target library is out-of-date with respect to the locally built library (or missing). In the bigger build picture, other objects dependent upon the library might avoid being rebuilt when an unnecessary library installation has been avoided. This is what is meant by "incremental" installation. Imagine the benefit we get from avoiding an unnecessary installation of stdio.h; a large amount of command rebuilding might be avoided. In practice, the above model can be complicated by a few factors: ROOTLIBRARY is permitted to be a list of libraries, ROOTLIBS, and the local object(s) '%' can be stored in an architecture-specific or machine-specific directory, MACH, when there are some specific sources to justify it. Furthermore, AT&T put profiled libraries in subdirectories named "libp," creating a special case. And, we have dynamic libraries. Consequently, the example becomes: LIBS= $(LIBRARY) MACHLIBS= $(LIBS:%=$(MACH)/%) PLIB= libp/$(LIBRARY) DYNLIB= $(LIBRARY:.a=.so$(VERS)) MACHPLIB= $(MACH)/$(PLIB) ROOTLIBDIR= $(ROOT)/usr/lib ROOTLIBS= $(LIBS:%=$(ROOTLIBDIR)/%) ROOTPLIB= $(ROOTLIBDIR)/$(PLIB) # install rules $(ROOTLIBDIR)/%: $(MACH)/% $(INS.file) $(ROOTLIBDIR)/libp/%: $(MACH)/libp/% $(INS.file) install: all $(ROOTLIBS) $(ROOTPLIB) (This example assumes the ROOT library directories are in place.) A macro, such as ROOTLIBS, representing a list of installed objects, fits a pattern-matching rule and thereby generates a list of dependencies which fit the right-hand side of the rule. These dependencies are predictable enough to be represented in another macro, such as MACHLIBS (or LIBS, if the source is machine independent). That macro is used as a target to build the dependencies of the "install" target. Being a list, it forms a list of targets with further respective dependencies. The list in this implementation supplants the for-loop we have previously associated with "install" targets. A for-loop is indicative of a non-incremental installation quite capable of triggering unnecessary secondary rebuilding; as such, it is to be avoided. The "all" dependency has been added to the install target for two reasons. First, the pattern-matching install rules fail to match in the event that the object to be installed has not yet been built, as would be the case, for example, if someone ran "make install" right after a "make clobber." Second, the "all" dependency forces the local targets to be up-to-date before the objects can be installed. Previously, mere existence was enough to satisfy a pattern-matching rule, sometimes causing out-of-date objects to be installed when they were newer than their old install targets. Incremental Builds ------------------ Incremental builds trade disk space for speed by retaining nearly all intermediate files from the build process. This is most noticeable with libraries built from regular objects, profiled, objects, PIC objects, or debug objects. The aim is to segregate these objects to prevent the building of one class of them from overwriting another previously built class. The implementation complicates build rules: the simple rule of placing a derived object next to its source has to be replaced with a set of rules which "know" how to derive segregated objects. In a sense, the rules know the class of an object by where it goes. Each class of object goes into its own directory. Because the directory names are arbitrary, they should be standardized by convention. In the example below, they are objs, profs, and pics. There are corresponding macro names to hold the respective lists of object names. OBJS= $(OBJECTS:%=objs/%) PROFS= $(OBJECTS:%=profs/%) PICS= $(OBJECTS:%=pics/%) The directories do not necessarily exist prior to a build, hence: objs profs pics libp: -@mkdir $@ Here is representative use of two of the macros and directories: ARFLAGS= r $(LIBRARY): objs $(OBJS) $(AR) $(ARFLAGS) $@ $(OBJS) libp/$(LIBRARY): profs libp $(PROFS) $(AR) $(ARFLAGS) $@ $(PROFS) where OBJS, PROFS, and other such dependencies can be built by: objs/%.o profs/%.o pics/%.o: %.c $(COMPILE.c) -o $@ $< The one rule suffices because COMPILE.c uses standard macros which can be assigned varying values dynamically: # conditional assignments $(PROFS) := CFLAGS += -p $(PICS) := CFLAGS += -K pic $(PICS) := CPPFLAGS += -DPIC If incremental builds are going to retain many intermediate files, the way to get the most benefit from the files is through the representation of intermediate dependencies. In the case of libraries, the installed library ultimately depends upon source files for its objects, but we retain the archive local to the source tree and all the object files it archives. If you rewrite one elaborate build rule into a series of rules and subtargets, keep a few things in mind. Retain the intermediate objects corresponding to the subtargets so "make" can stat them; don't remove them during the build process, you will lose the basis for incremental building. Remember to add the intermediate objects to the list of items removed by the "clean" target. If some intermediate objects very rarely change and have long-term value, associate them with clobber rather than clean. A series of pattern-matching rules works well with out-of-date intermediate files present. However, such rules can fail when the files are absent (as mentioned above in the justification for adding "all" as a dependency of the "install" target). The way to avert the failure is follow the pattern-matching rules in the makefile with explicit dependen- cies. It is crucial to get the ordering correct. An accurate representa- tion allows the build to resume work in small steps applying just the right rules. Here is a simplified example taken from the kernel's adb macro directory. SCRIPTS-sun4c is a list of targets to build in the sun4c directory. sun4c/%.adb.c : %.adb sparc/adbgen1 < $< > $@ sun4c/%.run: sun4c/%.adb.c $(BUILD.run) sun4c/%: sun4c/%.run $(BUILD.adb) # dependencies made explicit $(SCRIPTS-sun4c) : $$(@).adb.c $$(@).run Without the explict dependencies, "make" would not know how to build sun4c/foo from foo.adb in the absence of sun4c/foo.adb.c and sun4c/foo.run. With the dependencies, foo.adb yields sun4c/foo.adb.c that yields sun4c/foo.run that yields sun4c/foo. The above example serves to illustrate a chain of pattern-matching rules. However, it also illustrates a concept taken too far. In actual practice, people only changed the .adb source files. The intermediate objects were not independently changed, so the full chain of rules was fired in every instance. If intermediate objects do not change indepen- dently, then you are not gaining an incremental build advantage. You need to balance the overhead of additional rules with the advantage of accurate incremental builds from independent changes. Fast incremental rebuilds depend upon not oversimplifying the steps it takes to build each object and not overstating the dependencies of each step. Many opportunities for improvement remain in this area. Not all speed improvements stem from dependency information enhancements. In some build actions, commands can be replaced by appropriate makefile macros. For examples, architecture values can be derived from the predefined macro TARGET_ARCH, and $(@D) and $(@F) can replace "dirname" and "basename." For-loops can be replaced with suitable dynamic targets. There is choice among tradeoffs in representing dependencies. Overstated dependencies can slow the build down. Understated dependen- cies can break incremental builds or make them inaccurate. Let us study a configuration that presents these risks and tradeoffs. In some areas of the source base you can find several adjacent directories of command source and one directory of library source, where many or all of the commands depend on linking with one or a few libraries built in the library directory. For example, consider this diagram: topdir/ Makefile libdir/ cmddir1/ cmddir2/ cmddir3/ Makefile Makefile Makefile Makefile *.c *.o *.c *.o *.c *.o *.c *.o libfoo.a cmd1a cmd1b cmd2 cmd3a cmd3b There are several ways to build such a configuration; in selecting a method, you should consider build performance, incremental build accuracy, and completeness of the dependency graph. In general, our practices indicate we are willing to trade some completeness of the graph for speed in the build but not compromise the accuracy of incremental builds in the large. Three methods are worth consideration, each recommended within a range of configuration sizes. Viewed from the perspective of one of the directories of command source, the three methods are: 1) forcibly jumping to the library directory and checking each library against its sources, 2) merely checking the existence of each library and building each if it is missing, 3) not checking for the library, assuming it exists, failing otherwise. The essential details of (1) are: in top makefile: SUBDIRS = cmddir1 cmddir2 cmddir3 clean clobber := SUBDIRS += libdir all install clean clobber lint: $(SUBDIRS) FRC in a command dir: LDLIBS += $(LOCAL_LIB) $(PROG): $(LOCAL_LIB) $(LOCAL_LIB): FRC @cd $(@D); pwd; make $(@F) @pwd FRC: In a given configuration, (1) is the slowest to build because it is doing much redundant checking. It also represents the complete depen- dency graph, thus a build from scratch started in a command directory succeeds. The essential details of (2) are: in top makefile: SUBDIRS = libdir cmddir1 cmddir2 all install clean clobber lint: $(SUBDIRS) FRC in a command dir: LDLIBS += $(LOCAL_LIB) $(PROG): $(LOCAL_LIB) $(LOCAL_LIB): @cd $(@D); pwd; make $(@F) @pwd In the same configuration given above, (2) is markedly faster to build than (1) because (2) avoids redundant checking of library source, however it represents a truncated dependency graph. It indicates a dependence on the existence of a library, not necessarily an up-to-date library. A build from scratch started in a command directory will succeed, but an incremental build started there might use an out-of-date library since it checks only for existence. Hence (2) relies on the makefile at the top of the configuration to direct the build in the large into the library directory first, ensuring libraries are up-to-date before they are referenced. The essential details of (3) are: in top makefile: SUBDIRS = libdir cmddir all install clean clobber lint: $(SUBDIRS) FRC in a command dir: LDLIBS += $(LOCAL_LIB) In the given configuration, (3) is the quickest to build. The dependency graph omits explicit reference to libraries (although the KEEP_STATE mechanism keeps records of them). A build from scratch started in a command directory will fail because the libraries do not exist, so (3) relies on the top makefile in the configuration for the existence of the libraries. From these observations, the guidelines recommend the following. In general practice (2) is recommended because a small build in a command directory can jump over to a library directory and build a library if it needs it, but the expense of (1) is avoided. Although very small incremental builds may be inaccurate because of using out- of-date libraries, large incremental builds are correct because higher level makefiles will visit library directories before the command directories which use them. It is mandatory in (2) that the build in the large build libraries before they are referenced. For parallel "make" a .WAIT is necessary. In small configurations of source, having few commands and small libraries, you may want to use (1) for its completeness and accuracy. The small configuration implies a small performance penalty. With (1), the top makefile in the configuration should not descend directly into the library directory for "all" and "install" targets; it should leave the library build to be triggered as a side effect of the builds in the command directories. However, the top makefile should descend into the library directory for "clean" and "clobber" targets. In very large configurations (3) is recommended for speed in the build and less clutter in the makefiles. In (3) a failure of a top makefile to descend early into library directories will be obvious. For parallel "make" a .WAIT is necessary. Host Architecture Independence ------------------------------ Host architecture independence is implemented through an AT&T practice of using ROOT to specify a location, other than the build machine's root directory, into which the build process deposits header files and libraries for reference during later parts of the build. The default values of flags for build tools force references to be made under the directory specified by ROOT. Our default value for ROOT is /proto, but it is often overridden in builds in CodeManager workspaces, since workspaces often contain private proto areas reflecting local changes. Use of ROOT permits flexibility with regard to the release of operating system running on the build machine. The implementation is simplified by some complementary goals yet to be discussed, namely configurability and global flexibility; nevertheless the specifics are the same: the value of ROOT is used in all pertinent flags and preliminary directories and header files are installed under ROOT. (This material is taken from a number of makefiles.) ROOT= /proto CPPFLAGS= -I$(ROOT)/usr/include LDLIBS += -L$(ROOT)/usr/lib ROOTDIRS= $(TARGETDIRS:%=$(ROOT)%) ROOTLIBDIR= $(ROOT)/usr/lib ROOTLIBS= $(LIBS:%=$(ROOTLIBDIR)/%) # the Targetdirs file is the AT&T target.dirs file in a makefile format. # it defines TARGETDIRS and ROOTDIRS. include Targetdirs sgs: rootdirs userheaders libheaders sysheaders rootdirs: $(ROOTDIRS) $(ROOTDIRS): $(INS.dir) userheaders: FRC @cd head; pwd; $(MAKE) install_h libheaders: FRC @cd lib; pwd; $(MAKE) install_h sysheaders: FRC @cd uts; pwd; $(MAKE) install_h FRC: We have added the "ws" script to our use of CodeManager to set values of shell environment variables that specify ROOT references for include paths and load paths, automatically. For example, the specific value seen above for CPPFLAGS is replaced with a reference to shell variable ENVCPPFLAGS1. For referencing libraries, there is a similar variable ENVLDLIBS1. See the man page for the ws script. Suitably Global Flexibility --------------------------- Suitably global flexibility is implemented primarily with the makefile "include" mechanism and secondarily with great reliance on the make.rules file in /usr/share/lib/make. At the top of the source tree is Makefile.master, containing macro assignments of global importance. Changes to Makefile.master apply to the whole build. Here is a simplified version of the current Makefile.master: [ . . . ] # # Makefile.master, global definitions for system source # ROOT= /proto INS= install SYMLINK= ln -s CHOWN= echo CHGRP= echo FILEMODE= 644 DIRMODE= 755 OWNER= bin GROUP= bin INS.file= $(RM) $@; $(INS) -s -m $(FILEMODE) -f $(@D) $< INS.dir= $(INS) -s -d -m $(DIRMODE) $@ # installs and renames at once # INS.rename= $(INS.file); $(MV) $(@D)/$(