The Solaris Service Management Framework (SMF) [PSARC 2002/547 [SWAN:/shared]] includes the Service Configuration Facility (SCF), which is used primarily for configuration data storage, though it provides other functionality as well. In addition to internal usage, interfaces to SCF have been published so that end-users may utilize its functionality. Experience has shown that the facility's design did not include functionality required by certain users, which has compelled them to develop workarounds which are fragile and complex. Furthermore, the current design will hamper integration of new functionality required by future projects.

The rest of this section briefly explains how SCF is used today and highlights the problems in the current design and implementation. Section 2 (“Enhanced Profiles Model”) sets forth a new design which allows for better future functionality as well as more complete implementation of current functionality. Section 3 (“Enhanced Profiles Interfaces”) describes the interfaces through which the proposed new functionality will be accessible.

1.1. How SCF Works Today

1.1.1. Service Installation

SCF, also known as “the SMF repository”, is involved in all portions of the service lifecycle. The first phase of the lifecycle, installation, occurs when a system administrator installs a package which contains software intended to participate in SMF. The software developer indicates this intention to Solaris by including a file known as a service manifest, which directs the packaging system to deliver into a subdirectory of /var/svc/manifest. Service manifests are XML [Wikipedia:XML] files which specify the name of the service and its default SMF configuration data. During package installation on a live system SMF components read the manifest, create representative service objects in the repository, and add to those objects properties and values which represent the default configuration. For packages installed onto inactive Solaris root filesystems, this occurs during the next boot.

SCF properties have names, though since they are organized into property groups which also have names, they are usually referred to by a name pair, as in “general/enabled”. Properties are created with specific data types and can have any number of values of that type, including zero. The general/enabled property, for example, usually has type boolean and usually has a single value, true or false. Other legal types include count (unsigned integer), integer, time, astring (array of ASCII characters), and opaque (array of bytes). More types are defined as specializations of these base types.

The service manager, svc.startd, watches the general/enabled property of each service. If it has a single true value, then svc.startd considers the service enabled. svc.startd will start enabled services, but only after their dependencies, as described by other properties, are satisfied. svc.startd starts most services by executing a program indicated by the service's configuration properties.

The framework maintains status information for all valid services. Solaris administrators can retrieve this information through the SMF command-line interface, notably through the svcs command. The SMF components in charge of this information (usually svc.startd) communicate it by setting SCF properties on the appropriate service. For instance, the restarter/state property contains the service's state (such as “disabled” or “online”), and the restarter/state_timestamp property contains the time that the current state was entered. Publication via SCF makes the information available to developers through the svcprop command and the libscf(3LIB) interface.

1.1.2. Service Customization

To start or stop a service, SMF provides the svcadm enable and svcadm disable commands for the administrator. They modify the general/enabled property of the target service. Additionally, if the administrator wants to modify other aspects of how SMF handles the service, the svccfg command can be used to modify individual properties. Furthermore, if a service developer so chooses, the service software may be written to consult service-specific properties in SCF. If the service developer chooses to publish the name and meaning of the property, then administrators can modify the property's values with svccfg to reconfigure the service. Or, service developers can choose to keep the property names private and provide a service-specific tool which translates behavior-change requests from the user into appropriate SCF property modifications.

Other subcommands of the svcadm command are meant to control services, not to affect long-term behavior. svcadm restart simply restarts a running service. The request is transmitted to svc.startd by modifying an SCF property on the target service, though the property group it is contained in is marked nonpersistent. Nonpersistent property groups are discarded on shutdown, so the change is guaranteed not to have a long-term effect.

Historic copies of configuration can be invaluable. To this end SCF provides for snapshots of a service's properties. SMF automatically takes a snapshot of the configuration after a service starts successfully; the snapshot is named “start”, and since it represents a known-good configuration it can be useful after a configuration change causes the service to fail. Only SMF-initiated snapshots are available today; administrators are not allowed to take snapshots at arbitrary times.

One snapshot plays a particularly prominent role in SMF. The running snapshot is designated to contain the authoritative configuration values for the service — SMF guidelines instruct software developers to read configuration out of the running snapshot rather than from the properties directly attached to the service's representative objects. This is primarily because svc.startd automatically restarts services when their associated processes fail, and some failures, like a memory error, could happen at any time. If a service was restarted while the administrator was in the midst of changing its configuration, the service might read an inconsistent set of configuration values. Since the running snapshot is updated atomically, administrators can update it when their change is complete, and the service will either see the old version of the configuration or the new version, but never an inconsistent mix.

The running snapshot is updated by the refresh operation. Administrators can invoke it with the svcadm refresh command and software developers can use the smf_refresh_instance() function in libscf(3LIB).

At times users, or the framework itself, need to enable or disable groups of services. Accordingly SMF recognizes certain XML files which list services and whether each should be enabled or disabled as service profiles. Profiles may be applied to the repository with the svccfg apply command. svccfg extract will generate a profile which lists all of the services on the system and whether each is enabled or disabled. SMF uses profiles to enable services on a freshly-installed system. The default, generic_limited_net.xml, enables local services typically available on a Solaris system but disables all services which listen to the network, except for ssh. Solaris also delivers generic_open.xml, which administrators can choose as the default to enable all of the services which would have been enabled after an initial install of Solaris 10, including insecure network services. The administrator chooses generic_open.xml by setting the /var/svc/profile/generic.xml symbolic link to “generic_open.xml” before the initial boot.

Initial install also delivers platform specific profiles into the /var/svc/profile directory and sets the /var/svc/profile/platform.xml symbolic link to the platform specific profile for this system.

To summarize, SMF and SCF allow three ways to customize service configuration:

• service-specific tools (e.g., sharemgr)

• service-independent tools (e.g., svccfg & svcadm)

• profiles which represent high-level system characteristics.

1.1.3. Service Upgrade

When a service developer creates a new version of a service, it may use different configuration properties or may use the same configuration properties in a different way. The service manifest should be changed accordingly to deliver appropriate default configuration on initial installation. If an administrator upgrades from the old version to the new version, SMF detects changes in manifest files and attempts to upgrade the corresponding repository configuration appropriately. Specifically, it determines which properties have been customized by the administrator, preserves them, and replaces the rest with the new default values.

1.1.4. Service Removal

The final phase of the service lifecycle is removal. When an administrator removes the package which contains an SMF service, SMF components delete the corresponding service objects in SCF. svc.startd notices the deletions and terminates the services, if they were running. (SMF attempts to fail the package removal if the service is still running.)

The diagram below illustrates today's repository model.

1.2. Problems

Some problems involve developers specifying configuration.

• SMF developer guidelines say that aside from certain special services, manifests should specify that the services they describe be disabled (false for general/enabled). If a service should be enabled by default on a system, the guidelines state that the system profiles (generic_open.xml and generic_limited_net.xml) should be modified appropriately. Those files are kept in the source base of the O/N consolidation [opensolaris.org glossary], so developers of services outside that source base must coordinate changes across the two source bases. This is cumbersome, error-prone, and doesn't accommodate third-party software.

• svccfg apply, the mechanism which modifies configuration according to profile files, does not do so atomically. In fact, SMF does not allow atomic changes to properties in different service objects. This can lead to problems like 6531036 [SWAN: monaco] [opensolaris.org].

A lot of problems center around administrators customizing configuration and developers dealing with those customizations.

• The requirement to use svcadm refresh to commit changes

  • svcadm refresh is required to commit changes made with svccfg (for most properties)

  • svccfg doesn't inform the user that svcadm refresh is necessary, so users new to SMF don't know to do it.

  • Users who don't use svcadm refresh are often surprised that the changes don't take effect after a reboot, as is common with configuration files.

  • The only way to determine which services have uncommitted changes is quite tedious.

  In addition to usability problems, this architecture has systemic implications: A configuration-modifying program could encounter a fault after making changes but before committing them. What should another configuration-modifying program do if it encounters uncommitted changes? It shouldn't delete them since they could be the beginning of a set of changes from the administrator. It shouldn't commit them since it doesn't know if they're complete. All it can do is notify the user that uncommitted changes exist. Of course, the user may not be able to tell whether the change is complete and may not be able to recover from the original fault.

• Though configuration in SCF can be customized with svccfg, and to a limited extent svcadm, management of those customizations after-the-fact is incomplete. Finding what has been customized in the repository is difficult — doing so through either the library interfaces or the command-line interface is not well published and tedious or error-prone at best. An enterprising administrator might be inclined to work around this by putting all of his customizations into a profile and applying it, but profiles don't allow properties besides general/enabled to be set. And even then there is no way to undo the application of a profile.

• SCF-resident configuration is problematic for administrators who need to install multiple systems with common customizations (i.e., JumpStart installation [docs.sun.com]). The published SCF interfaces are “live” — libscf(3LIB), svcadm, and svccfg all operate on the currently running system. The only customization of an inactive system allowed by SMF is to use the site.xml profile, which can only specify services that should be enabled or disabled. This is considerably less configuration power than with more traditional configuration stores (e.g., plain files). Moreover, administrators will encounter this problem more frequently as more service-specific configuration is stored in SCF, per Solaris advice.

• A goal of the SMF team in implementing SCF was to always preserve the administrator's customizations, even across upgrade. This is an admirable goal, but SCF doesn't empower software developers to maintain it when configuration should be changed programmatically. For example, an instance of the wpad service should be started when a wireless network interface is configured for WPA security. dladm should set the general/enabled property of the respective service to true, unless that property has already been customized by the administrator. But SCF offers no direct way for dladm to tell whether this is the case, so it proceeds unconditionally.

  On the flip side, because there is no way to tell whether a property has been customized, the svcs command must always guess that when a disabled service is causing problems, it was disabled by the administrator, which often isn't true.

• Most configuration should be read from running, but some shouldn't. Which to do has to be hard-coded into the reader, which is unsavory architecturally. Similarly, nonpersistent configuration doesn't appear in running at all. svcprop makes an attempt to hide this from the user, but can't entirely.

• Service status and actions are not read-only which allows the administrator to change their property values. This doesn't lead to a service failure, but may be disconcerting to see the value of restarter/state to be "foo" instead of "online".

• The Secure By Default (SBD) project sought to define a new set of default configuration — one which minimizes network vulnerability. The project team also wanted to offer an “open” configuration, as was installed by default on Solaris 10 and before. SMF profiles seem like a natural fit, but they are currently only allowed to specify services to be enabled or disabled, whereas the project needs to also modify other properties. As a result, the project introduced the netservices command, which employs profiles and manipulates the aforementioned configuration. Once a secure by default mode is chosen, the repository doesn't provide a way for the administrator to audit the configuration later, to observe what has changed.

• The NWAM project will need to be able to set configuration based on the network environment. Today that would require a large amount of accounting data outside the repository. That could be eliminated with a restructured repository.

A few problems are specifically around the upgrade process.

• An administrator can't specify that the default value for a configuration property should be used, but it should be treated as a customization and not replaced in the future. This is known as the “customize-to-same” problem.

• A side effect of the incomplete upgrade mechanism causes great consternation amongst service developers who wish to deliver an alternative instance of a service delivered by another developer (most commonly, Sun). This can happen when the original manifest describes the properties of a service. When a different manifest is added to the system that contains the same service where the properties are not associated with the service but instead are associated with a new, non-default instance, the import of the second manifest causes SMF to assume that this is an upgrade of the original manifest. The upgrade of the service will remove the properties originally associated with the service. This problem is exaggerated because the current convention within Sun is to deliver most, if not all, properties at the service level.

• Another side effect of the upgrade mechanism is that if a developer delivers a service with no instances, as is common for developers who wish to create instances dynamically, then the properties delivered by the service will not be upgraded automatically.

• If a developer's service should be enabled by default, then he can follow SMF's convention to modify a system profile to enable that service. But these profiles are ignored on subsequent upgrades. This is because SMF can't tell which services are enabled or disabled because of the initial profile application and which services are enabled or disabled because of the administrator's intent (i.e., a customization). Therefore SMF opts not to risk undoing administrative customizations, and doesn't reapply profiles after an upgrade. To enable the service on an upgraded system, the private and tedious /var/svc/profile/upgrade file must be used. Essentially, profiles aren't upgradable.

• If a service developer creates a new version of his service which interprets configuration differently than the previous version, then the upgrade procedure should, if possible, migrate customizations to preserve administrative intent, or inform the administrator when this isn't possible. The current upgrade scheme leaves customization identification up to the service.

• When services should only run on certain platforms, developers are advised to modify the appropriate platform profile. However, like the generic profiles, platform profiles are not reapplied on upgrade. Additionally, if a new, more specific platform profile becomes available, it is not applied, either. Plus, if an image is booted on different platforms (e.g., for Xen), the subsequent platforms will not be detected properly.

We propose two enhancements to SMF profiles:

1. Profiles be represented in the repository.

2. Profiles be allowed to contain properties other than general/enabled.

2.1. Concepts

2.1.1. Repository Profiles

The central problem in configuration upgrade is the treatment of customizations. Software developers expect that uncustomized configuration be replaced with new default configuration. Administrators expect that configuration customizations (or more precisely, their intent) are preserved. SCF's current customization-handling is minimally functional, but produces a number of undesirable side effects and isn't amenable to extension.

We propose to replace SCF's current customization-treatment strategy (detect customizations at upgrade-time through the use of snapshots) with a new one: distinguish default configuration from customizations by keeping them separate. Then default configuration can be upgraded by replacement, without disturbing customizations.

Profile files already embody a limited form of customization separation in SCF, so we propose to expand the concept both to contain arbitrary configuration (not just values of the general/enabled property) and to govern configuration structure within the repository. In particular, all configuration values will be contained within some profile.

When multiple profiles contain values for the same property, the repository must choose which to present to readers. Clearly customizations should be preferred over defaults. To allow customizations and defaults to be subdivided into more meaningful profiles, we chose to implement this with a stacking algorithm. That is, profiles will be ordered and for a given property the “highest” profile which contains values for it wins. (More on ordering later.)

Suppose the default configuration prescribed by service developers (via service manifests) were placed in a profile named base, and administrative customizations were placed in a profile named local. Then the profile structure in the repository of a freshly installed system might look like

In the diagram profiles containing default values are blue and profiles populated by administrator customizations are orange. The platform_sun4v and generic_limited_net profiles represent the platform_sun4v.xml and generic_limited_net.xml profile files used today. Ordinary configuration modifications made with svcadm or svccfg would be placed in the local profile.

Creating a separate in-repository profile for generic_limited_net will solve the upgrade problem for developers who add new enabled-by-default SMF services. Instead of working around the fact that generic_limited_net.xml isn't reapplied on upgrade by adding code to the upgrade file to enable the service, modifying generic_limited_net.xml to enable the service (as required for initial install) should suffice since the upgrade procedure should replace the in-repository generic_limited_net profile's contents with the new contents, and the service will automatically be enabled. Of course, if the administrator has explicitly disabled the service in local, that customization will take precedence.

Profile Order

When determining the value of a property, profiles containing administrative customizations should be searched before profiles containing system defaults; otherwise, the default values would always override the customizations effectively negating the administrator's choices.

As shown above there may be several profiles describing administrative customizations such as the local profile and the generic_limited_net profile. Since the local profile contains customizations made by the administrator via svccfg and svcadm, the local profile is always active on the system. The generic_limited_net profile was chosen to be active when the administrator installed the system to have networking enabled with SBD. The administrator could also choose to open up network services causing the generic_open profile be active instead of the generic_limited_net profile.

The platform_sun4v profile was not chosen through any administrative action, but was activated by the system when running on a sun4v platform. The base profile was also activated by the system and represents the service manifests installed on the system. Even though the administrator can make choices during installation that may alter the service manifests installed on the system, the system adds the service manifests into the base profile and this cannot be modified by any administrative customizations.

The organization of profiles in the system seems to fall into two groups: administrator initiated and system initiated. In order to manage the order of profiles between and within these groups, each group will have an precedence level that contains the order of active profiles for that group. The precedence levels will be named admin and system describing if the profile entries in that level are a result of an administrator or system action.

Figure: Precedence Levels

The admin precedence level contains entries for profiles that were activated by an administrative choice. If the user chose an open network configuration then the generic_open profile would be in the admin precedence level. If the user chose to limit the network configuration then the generic_open entry would be removed from the admin precedence level and would be replaced by an entry for generic_limited_net. The local profile is shown at the top of the admin precedence level since it will be searched first when determining a property value. This allows the administrator to override a property value contained in either of the networking profiles.

The system precedence level contains entries for profiles that were activated by the system. If a search for a property value wasn't found in the list of active profiles in the admin precedence level, then the profiles listed in the system precedence level will be searched. The platform_sun4v profile will be searched before the base profile. This allows sun4v-specific services to override settings in the base profile. The base profile is always active and will be the last profile searched when determining a property value. The general/enabled property of most services will be disabled in the base profile.

Non-configuration data

In addition to configuration, SMF stores other data in SCF. Status for each service, such as the current state, is published in properties of the service. Status is primarily consumed by the administrator through commands such as svcs, but developers may write programs which use it as well. SMF also uses properties to transmit user requests; svcadm restart sets a certain property on the target service and svc.startd restarts the service upon detecting the change.

Consumers of status and user requests rely on accessing the most recent data set by the producers. If this data were placed in a profile which was below the administrative customization profiles (i.e., the admin precedence level), an administrative customization would obscure all updates to the properties and break functionality. Therefore SCF will provide a precedence level above admin, called system-override, for this data.

The SMF_status profile will exist at the system-override precedence level and SMF restarters will set properties in this profile. If a sly administrator sets an SMF status property to a different value in a lower precedence level profile, this has no effect since the profiles are always searched starting with the highest precedence level.

For communication of user requests, SCF will provide the SMF_actions profile in the system-override precedence level.

SCF will provide the generic_status profile at the system-override precedence level for service developers to use to publish status.

Data written to SMF_status, SMF_actions, or generic_status will be treated as nonpersistent, and will be discarded on system shutdown.

Immutability

The proposed upgrade procedure calls for upgrading developer-delivered profiles by replacement. For administrators who place customizations in such a profile, this will result in a truly lousy experience. To prevent this, the repository will allow profiles to be marked immutable, which will cause attempts to modify individual properties in them to fail. In the example above, base, platform_sun4v, and generic_limited_net would be immutable. Of course, sections of immutable profiles must be replaceable for upgrade to work at all.

The Completeness Flag

Today when svccfg apply changes properties according to a profile file, it skips over services which do not exist. If in-repository profiles behaved this way then

1. applying a profile with customizations for a service which doesn't exist and

2. creating the missing service

would result in an in-repository profile different from the profile file. To avoid this inconsistency, the repository must remember profile customizations even for services which do not exist, and those customizations must be utilized as soon as the service is created.

Storing configuration for services which don't “exist” calls into question the existence criterion. We contend that the crucial attribute is completeness. Configuration delivered by a service manifest into the base profile is known to be complete and may be used by the framework to start the service. Configuration delivered by a profile is usually only intended to modify configuration and doesn't make sense on its own.

A simple way to differentiate would be for services to only exist when they have configuration in the base profile. Unfortunately, this test is wrong for two cases.

Service dependents

SMF allows the developers of a service (say, provider) to declare another service (say, consumer) to be dependent. In other words, consumer should depend on provider. (This is also known as a “reverse dependency.”) If this configuration were confined to provider, then to find all of the dependencies of consumer, a program would have to examine the configuration of all services on the system, looking for dependent configuration. Instead, the SMF team made the tremendously insightful decision to represent dependents by creating dependency configuration in the dependent service. (In this example, a dependency on provider is created in consumer.)

Since the external dependency is delivered by a service manifest, it will be placed in the base profile. The configuration represents a modification to the dependent service's configuration, however, and should not be interpreted as complete configuration.

Dynamic instances

Today programs can create service instances at runtime. (ksslcfg(1M) does this.) In addition to the base profile not being an option due to its immutability, instances that represent administrative customizations should be placed in a profile in the admin precedence level.

A better heuristic may exist, but the configuration writer should know whether its material is complete. Thus SCF will allow writers to set a completeness flag for service instances in profiles. svccfg import would set it for services created from service manifests, and SCF can treat service instances without it as nonexistent.

Multi-Package Profiles

Like generic_limited_net, profiles will often affect multiple services. To eliminate source base coordination problems, as explained above, we propose that profiles accept property values from multiple packages. Since files are the smallest unit of package delivery, this requires that profiles can be created from multiple files.

This can be implemented by allowing profiles to be created from profile directories, which will be directories of change manifests. Change manifests will be files which prescribe properties and values for service and instance objects. The specific format will be described in the Interfaces section.

With this capability, delivered property values may be restricted to the package which delivers the service. This makes for a more accountable system, and so we will mandate it. It's difficult to enforce automatically, though, so it will be a documented guideline for developers.

2.1.2. Conditional Configuration

Before the administrator customizes any configuration, some developers wish for their service to be enabled in certain scenarios and not others:

• svc:/system/dbus:default should be enabled by default in the global zone, but is a waste of resources in nonglobal zones (since hald isn't delivered into nonglobal zones).

• The svc:/network/name-service-cache:default service should be enabled by default, except in nonglobal zones on Trusted systems. (Something about “multilevel name service”.)

• The ipmid daemon (not yet integrated) should only be enabled when a bmc device is present.

SMF lacks formal support for this; developers persevere by always enabling the service by default, adding functionality to the service to detect when it should not run, and coding the service to disable itself in that case (usually temporarily). This is undesirable for two reasons. First, if the administrator explicitly enables the service in a scenario where it thinks it should not run, the service will automatically become disabled, which will likely be confusing to the administrator. Second, with in-repository profiles SCF will interpret the service's disable operation as an administrative customization, which is not accurate.

We propose to support these non-constant default policies with conditional configuration: SCF will allow developers to deliver condition objects and profiles with predicates. Condition objects will have a name and a nonpersistent boolean value. Predicates will be boolean expressions of condition object names. When a profile's predicate evaluates to false with respect to the condition values, SCF will consider the profile to be inactive and will not present its values to readers.

With this mechanism, the dbus developers could deliver (1) a service manifest which prescribes a default value of false for system/dbus's general/enabled property, (2) a condition named SUNW_global_zone, and (3) a profile named enable_system_dbus (which contains a true value for system/dbus's general/enabled property) with a predicate of “SUNW_global_zone”. Then if SUNW_global_zone is true in global zones but false otherwise, system/dbus will behave as desired without enabling or disabling itself. Furthermore, the administrator can permanently override this default policy in a profile at a higher level (e.g., local), or even override the policy only under certain conditions (by creating a profile with a predicate). Of course, a condition with a name like SUNW_global_zone should really be delivered by core Solaris, and the dbus developers should use a condition which more precisely reflects the meaning they require, like SUNW_hald_available.

Leaving how condition values are determined for later, note that this is also an ideal mechanism for selecting the appropriate platform profile. SMF's current special-case platform profile application code can be replaced by conditions for each platform (e.g., SUNW_platform_SUNW_Sun-Fire) and appropriate predicates on the platform profiles.

This would also be very useful in accommodating NWAM. For each recognized network environment, NWAM could create a condition object, such as SUNW_NWAM_home. NWAM configuration tools could place user customizations intended for each environment in a separate in-repository profile and set an appropriate predicate on those profiles. When NWAM recognizes a network environment, it need only set the corresponding condition object to true and SCF will automatically enact the user's desired customizations — atomically, even. Since in-repository profiles would not be NWAM-specific, generic tools could view, manipulate, and even predict configuration on the user's behalf, without NWAM-specific knowledge.

Thus the profile stack on an NWAM system might look like:

Condition values will be settable through SCF developer interfaces. We believe, however, it will be common for conditions to represent a constant attribute of the current operating system instance. To save developers from creating and delivering an SMF service which evaluates the condition, such developers will be allowed to deliver just the initial evaluation logic. One way to do this would be with an executable which reports the condition value when svc.configd invokes it at an appropriate time. Another way would be to expose the evaluation results in the kernel, perhaps via a kernel module. Both of these possibilities are fraught with a number of ancillary issues. Since the required conditions will be lightweight to evaluate (e.g., determining whether execution is taking place in a nonglobal zone, or the current hardware platform), we propose that for the initial delivery of this project the requisite conditions be hard-coded into svc.configd. Design and implementation of interfaces which allow developer-delivered attribute conditions will be left for a subsequent project.

Service Level Profiles

The ability to enable network services conditionally on the presence of a network raises the question of what configuration should be effective in the absence of all networks. If generic_limited_net were permanently active, as shown above, the network/ssh service would be enabled when there is no network. On a freshly installed system, this would be a waste of resources. (Not that we will prevent the administrator from enabling ssh under such conditions anyway.) generic_limited_net should not be deactivated, however, since it enables important system services like syslogd.

This problem stems from generic_limited_net's influence on services with different network usage and security characteristics whereas NWAM provides a mechanism to adjust service level and behavior based on network availability and identity. Therefore we propose that generic_limited_net and generic_open be refactored along these boundaries. We identified four classes of services in the profiles:

• Services which are important to the system.

• Services which offer service to local users.

• Services which offer service to remote users (i.e., the network).

• Services which offer service to remote users, in a secure fashion.

We propose that Solaris deliver three profiles which enable separate classes of services: local_services, net_services, and secure_net_services. System services should be enabled by default in the base profile.

Note that some services, like sendmail, can offer service to both local users and remote users, or only local users. If this can be controlled by an SCF property, then local_services should prescribe local-only mode, and net_services should prescribe “open” mode.

Configuration Sharing

With NWAM profiles, a significant number of users will want to create configuration profiles which contain most of the configuration in other profiles, but with a few customizations. For example, on his home network a user may wish to enable traditional network-facing services, but with network/telnet disabled. As proposed, the user could create a profile called “home”, copy the contents of generic_open into it, and then change the general/enabled property of network/telnet:default to false. This would work, but if the generic_open profile was changed during an upgrade, the home profile would not be changed in sympathy. A means for the user to express the intention that home be the same as generic_open plus a few customizations is missing.

We propose to allow profiles to include other profiles. Instead of copying the contents of generic_open into home, the above user would include generic_open in home. To customize whether telnet is enabled, the user can place the customization in the home profile, which will always take precedence over the property values in the included profiles.

This mechanism also allows parts of more stable profiles to be made conditional. For instance, the system/dbus service isn't important for system functionality, but we do want it to be enabled for users. So we recommend that instead of being automatically enabled in the admin precedence level, it should be included in the local_services profile, so that if the administrator removes local_services from admin, system/dbus will not be automatically enabled.

Conditional Profile Order

Being system-chosen, profiles such as platform_sun4v should be activated in the system precedence level. By virtue of holding administrator customizations, NWAM profiles should be activated in the admin precedence level.

Consider, though, if the platform_sun4v profile, which enables platform/sun4v/drd, were placed below the base profile, which disables platform/sun4v/drd. Then when platform_sun4v's predicate was true, the effective configuration would not change because the base profile would always override it, thereby defeating the profile's purpose. We conclude that conditionally active profiles should always reside above permanently active profiles. In essence, the permanently active profiles provide a substrate of default configuration upon which conditional profiles enact context-specific exceptions. (Maybe this isn't always true. See open issue “Permanent profiles to override conditional profiles.”)

The relative order of two conditional profiles is irrelevant if they contain values for disjoint sets of properties. If they do overlap but their predicates are never true at the same time, then their order is also irrelevant. Order is only significant for two overlapping profiles which can be active simultaneously. Clients could be allowed to specify an order between conditionally active profiles, but we believe that specifying different values for the same property under conditions which may coincide is ambiguous, and an error. Instead the user should determine which value should be used when the predicates coincide and should expand the predicates appropriately. (This is always possible with boolean expressions: a negating term can be added to the predicate of the undesired profile.) Therefore SCF will not allow users to specify a placement for conditionally active profiles, and will activate them in essentially random order.

An additional reason for not allowing users to specify order placement for conditional profiles is the complexity required to correctly specify the placement while conditional profiles are automatically activated and deactivated based on changing system conditions. The administrator would have needed to exhaustively define where the conditional profile should be activated based on what other conditional profiles may or may not be active at that time.

Random ordering of conditional profiles may be just dessert for developers who deliver ambiguous configuration, but it could frustrate administrators who configure profiles interactively. It's possible to determine programmatically whether predicates could coincide, though there would be many false positives without knowledge of which conditions are mutually exclusive. Therefore this project will provide a mechanism for condition developers to declare mutual exclusivity, and it could deliver functionality to determine when predicates might coincide. User interfaces would use the latter to preemptively detect when profile conditions are ambiguous and guide the user to a resolution.

This project won't deliver any such sophisticated user interfaces, however, so we will leave implementation of ambiguity-detection functionality for a future project. (See future work item “Ambiguous conditional configuration detection.”)

2.1.3. Backwards Compatibility

Naive Interface

We will refer to the current libscf(3LIB) functions and the service objects, instance objects, property groups, and properties accessible through them as the “naive interface”. All clients of today's repository are “naive” and non-administrative clients are expected to remain that way.

The data-reading functionality of the naive interface will remain the official interface for consumers of property values. The data-modifying functionality will also remain, but it lacks a channel to communicate the type of data and how long it should be effective. Thus such changes must be assumed to be configuration customizations, and clients which intend to supply other types of data, or data which shouldn't last forever, must use the profile-aware interfaces.

As such, it's straightforward for modifications from the naive interface to be placed in the local profile. But that wouldn't make sense for a property overridden by a conditional profile, say home. It's tempting to say that the overriding profile should be modified, but it could be immutable or the value could be deactivated in the future, which would be different semantics from today's repository, where modifications are to be kept forever.

Therefore we propose that the local profile be placed above the conditionally active profiles in the admin precedence level. This could be done by special-casing local, but we think it would be cleaner to introduce a new precedence level named admin-override above the admin precedence level, for the local profile. Then modifications through the naive interface will always take precedence over any conditional profiles in admin.

The admin-override precedence level must be above admin, but it should still be below system-override. As such, values of some properties visible through the naive interface may come from profiles above local. Notably, the non-configuration profiles such as SMF_status and SMF_actions. Since these shouldn't be customized anyway, attempts to modify such properties through the naive interface will fail. This is a change of behavior, but we think it is warranted since those properties were never intended to be modified by users or programs outside of the O/N consolidation anyway.

Moving local to admin-override leaves no generic location for customizations which should be used when no condition overrides them. For this we propose that the repository provide the local_default profile, unconditionally active in the admin precedence level.

Dynamic Instances

Since svccfg can create incomplete service instances to hold dependency configuration, svc.startd must employ a heuristic to determine when a service instance is complete and can be started. If svccfg creates such instances in profiles without setting the complete flag, then those instances won't appear in the naive interface and svc.startd will no longer need the heuristic.

Today programs can create empty service instance objects with the scf_service_add_instance() function, however. To maintain backwards compatibility, the corresponding instance object in the local profile must have its complete flag set, even though the instance has no configuration. Therefore svc.startd will need to retain its heuristic.

To simplify this in the future, however, we propose to deprecate the scf_service_add_instance() function, at least in order to create dynamic instances. Clients should instead create the instance with the profile-aware interfaces, which will allow them to create and configure instances atomically.

Configuration Change Model

SMF's current configuration change model is centered around two principles:

• Services should read their configuration not from the properties contained by their corresponding SCF service and instance objects, but from the running snapshot of those properties.

• The running configuration snapshot of a service is updated during a “refresh” operation on that service, which also notifies the service that the configuration may have changed.

This model is problematic for automatic configuration changes, as proposed for NWAM. If the network environment changes, the user's desired configuration profile should be activated and the new configuration values should be used by the framework and running services. That requires the changes to be propagated to the running snapshot, which in the current model requires a refresh operation. If, however, the administrator is in the midst of changing configuration when NWAM detects an environment change, such a refresh might push only part of the user's configuration changes into the running snapshot, and it's likely that the resulting configuration will be inconsistent. The solution is separation: it must be possible for changes prescribed by a condition change to be committed to the running snapshot separately from changes prescribed by the administrator.

We propose that SCF with enhanced profiles provide a buffer for administrative changes and immediately apply condition-triggered configuration changes to the running snapshot.

The buffer used for administrative changes will be a special profile called “editing” at the admin-override precedence level. It will always be active directly above the local profile. Property modifications made through the naive interface will be placed in the editing profile, where they will be visible through the naive interface, as today. But those changes will not be visible in the running snapshot, which will essentially be a view of the service's properties from the local profile down. The refresh operation for an instance will be enhanced to flush the associated changes from the editing profile into the local profile, where they will become visible in the running snapshot.

Changes to profiles from local down, however, will be visible both through the naive interface and the running snapshot. This automatic refresh operation will occur when these profiles are activated/deactivated by either a direct administrative action or by an induced condition-triggered configuration change. But, the automatic refresh would be problematic for configuration modifiers which need to modify multiple property groups in a particular profile consistently, since the current interface only guarantees transactionality on individual property groups. Therefore this project will introduce interfaces with larger transaction domains. If a modifier can change all of the properties of a service instance atomically, then there is no need for the separate refresh operation to commit the changes to the service-visible running snapshot.

When a profile is activated or deactivated below local, the running snapshot automatically reflects the changes contained within the profile. The repository will notify svc.startd that each service instance described in the profile needs to be refreshed. The refresh will cause the service instance to re-read its property values from the repository using the running snapshot.

Milestones

Solaris administrators can use runlevels to manage services system-wide. To change the runlevel, certain services should be started or stopped. SMF implements runlevel transitions with milestones. To “transition to a milestone”, svc.startd starts or stops services according to their dependencies.

During a milestone transition, svc.startd stops services by temporarily disabling them. It cannot temporarily enable services which should be started, however, lest it start services which the administrator never wanted to run. Instead it deletes any temporary override to the service's general/enabled property so that the service will only start if it is enabled persistently.

Due to the backwards-compatible naive interface, this would continue to work without modification. This project, however, presents an opportunity to improve the situation. Today svc.startd destroys information when changing the milestone — if the administrator had temporarily disabled a service, then that temporary override will be deleted if the runlevel is taken down and brought back up. Also, since the milestone setting has to override even the local profile, svc.startd would have to make changes either in a profile above local, or in local directly. The latter would be regrettable because SMF tools wouldn't be able to determine whether the user had disabled a service generically (e.g., with svcadm disable), or through a runlevel or milestone request.

We propose that svc.startd change the service milestone by creating a profile called “milestone” and activating it above local. In it svc.startd will place property value overrides which temporarily disable the services which should be stopped. Though the contents of this profile is set by the system, its presence is an administrative customization, so we believe it should be placed in the admin-override precedence level, above local.

As stated above, properties overridden by profiles above local will not be modifiable through the naive interface. This, however, would regress behavior in a restricted milestone. Today after a service has been disabled by a milestone transition, the administrator can still enable or disable it, which can be useful for testing. To preserve this, the milestone profile requires special behavior: punch-through. That is, if a client attempts to modify a property through the naive interface, but the value for that property is currently coming from the milestone profile, then SCF will delete the override in milestone and will make the modification in local. In a sense, milestone is ephemeral, or a point-in-time modification.

Punch-through behavior could be introduced as an independent attribute of profiles that can be turned on or off, but it is really only valid for profiles above local. Therefore SCF will only allow mutable profiles to be unconditionally activated in admin-override above local, and they will automatically be subject to punch-through behavior.

2.2. Notable Use Cases

For concreteness, this section will discuss intended use cases. Also, a few use cases have motivated additional features.

2.2.1. Secure By Default

The Secure By Default project [@opensolaris.org] modified the default configuration of a freshly-installed Solaris system to minimize network exposure. It also provided the netservices command, which allows the administrator to change configuration to an “open” status, or back to the initial “limited” mode. Furthermore, the administrator could select the “open” configuration between initial install and initial boot by setting the /var/svc/profile/generic.xml symlink to “generic_open.xml”.

If services which are not important to the system or the platform are disabled by default, then instituting SBD's “limited” configuration is a simple matter of not activating the net_services profile.

After initial install, the “open” configuration can be effected by activating the net_services profile. There are a few services, however, which should remain enabled across the change from “limited” to “open”, but should change behavior (such as opening sockets to listen on the network). A few of those services are not capable of changing behavior dynamically, and must be restarted. Profile activation won't restart services, so the netservices command will have to be retained.

After setting a particular mode, administrators will want to verify that the configuration is still in force, and determine what customizations have been made. For this, the project will provide profile verification functionality. Supposing it is exposed through svccfg verifyprof, then svccfg verifyprof generic_limited_net should report the differences between the currently effective configuration and that prescribed by the generic_limited_net profile. Similarly for the generic_open profile.

Furthermore, with conditional configuration, the user could choose open mode for only certain scenarios by activating net_services conditionally. This would be useful with NWAM profiles. It might make sense, therefore, to allow users to use verifyprof with a hypothetical set of condition values. However, until services which should change behavior dynamically during a mode switch can do so, we will advise against conditionally activating net_services since automatic transition from an “open” mode to a “limited” mode (without netservices) would leave the processes of those services in “open” mode despite SCF configuration reflecting “limited” mode.

It's important to note that it's nearly impossible to enforce third-party contributions to the net_services and generic_limited_net profiles. Therefore the above guarantees can only apply to Sun-delivered services. Every effort needs to be made to communicate this to users, including documentation and perhaps naming conventions, so they don't expect the guarantees to extend to the third-party services.

2.2.2. wpad

The changes described by PSARC 2006/046 (Wireless WPA Supplicant) [SWAN:/shared] [opensolaris.org] include the wpad daemon, which handles WPA [Wikipedia] encryption for network interfaces. The rub is that wpad can only handle a single interface, so a single svc:/network/wpa:default service isn't sufficient — there must be a separate service instance for each interface using WPA encryption.

Today the project could deliver a svc:/network/wpa service with no instances, and when WPA encryption is enabled for an interface, libwladm can create an instance of network/wpa for that interface and temporarily enable it.

Service instances are always persistent, however, whereas these instances represent hardware which isn't guaranteed to be present after a system shutdown. If the user removes an interface from the system then a wpad service instance would exist for an interface which doesn't, which is a waste of resources and potentially confusing. This could be solved by allowing enhanced profiles to be nonpersistent, and by creating the instances corresponding to the hardware in such a profile.

We believe such a profile should not be placed in the admin precedence level because the customization isn't portable — it's tied to the hardware. Therefore we recommend that the WPA developers activate the profile in the system precedence level.

2.2.3. JumpStart

The functional nature of SCF's public interfaces has limited administrators' ability to configure systems before they have booted for the first time, which is the case during JumpStart. Currently customization of configuration resident in SCF is limited to /var/svc/profile/site.xml, which can only enable or disable services. Enhanced profiles should allow administrators to customize any property. Administrators will need a way to control the new attributes of profiles introduced by this project, such as activation characteristics.

We'll offer an executable which may be used to specify which profiles should be activated on boot, and precisely how. The functionality available through this interface will be quite limited, though, since commitment constraints will prevent it from reading the repository datastore.

2.2.4. Virtual Consoles

The virtual console project [opensolaris.org] will introduce new devices to represent the virtual consoles of the system. Users will expect ttymon's to run on some of them by default.

ttymon is already run for the system console by the svc:/system/console:default service. For consistency, ttymon's on virtual consoles should be run as alternate instances of the system/console service.

Assuming that virtual console devices can be enumerated, we recommend that the developers introduce a service which creates instances of system/console for each virtual console device. This should be done in a nonpersistent profile which is unconditionally activated in the system precedence level. As the service creates the instances, it can specify the default configuration for each, including whether they should be enabled or disabled. The administrator can subsequently customize which are enabled or disabled as with any other service.

This scheme should also work if virtual console devices are allowed in nonglobal zones.

There has been discussion of starting ttymon's on demand. To effect this, the setup service could also create condition objects for each virtual console and create a profile which enables the corresponding ttymon when the condition is set to true. Then the code which determines that a ttymon should be started can set the appropriate condition to true.

2.2.5. Duckwater

Duckwater will allow the user to associate name services with each network environment. The Duckwater tool will allow the administrator to set name service properties in profiles that will be activated in conjunction with the NWAM profiles. For this reason, the name service properties could be placed directly into the corresponding NWAM profiles.

If the user had the same name service properties in multiple NWAM profiles, the user should be able to pull those common properties into a single profile which would be included by the NWAM profiles.

2.2.6. Profile List

After an initial install or upgrade to Solaris with Enhanced Profiles, the system will activate certain profiles. Before NWAM utilizes enhanced profiles, it will result in

These profiles will also be present on the system but will not be activated by default:

If the user requested an open configuration during install (see SBD), the net_services profile will also be added to the admin precedence level, above the local_services profile.

2.3. Details

This section explains the details of the model. The diagram at the end of this section may be helpful.

2.3.1. Conditions

SCF will have objects called “conditions”. Each will have a name, a boolean value, a stability, a description, an optional group name, and a set of authorization strings. Conditions may also be marked nonpersistent, in which case they will be discarded on system shutdown. When a condition is created, its initial value will be false. When the system shuts down, all condition values will be reset to false.

All conditions with the same group name will be considered mutually exclusive. Setting the value of a condition with a group name to true will automatically (and atomically) set the value of all other conditions with the same group name to false.

SCF will provide interfaces for developers to change the value of a condition. Value changing will only be allowed by processes owned by a user which possesses either the solaris.smf.modify authorization or an authorization associated with the condition.

Only users with the solaris.smf.modify authorization will be allowed to create condition objects.

The condition names “true” and “false” will be reserved by SCF.

2.3.2. Precedence Levels

The repository will have objects called “precedence levels”. They will have a name and will contain references to profile objects (described below).

Each profile reference may have a predicate associated with it. A predicate is a boolean expression of condition names (including the operators &, |, !, (, and )). References to conditions which do not exist will be evaluated as false. Without a predicate, the profile's predicate will be used, if it exists. The order of profiles with constant predicates or no predicates will be significant. Conditional profiles will effectively be kept at the top of the list, in an undefined order.

Precedence levels will be static and cannot be created or destroyed. There will be four of them: system-override, admin-override, admin, and system.

The system-override precedence level will always contain nonpersistent mutable profiles named SMF_status, SMF_actions, and generic_status. Conditional profiles may not be added to the system-override precedence level.

The admin-override precedence level will always contain mutable profiles named editing and local, in that order. Profiles may not be inserted between editing and local. Conditional profiles may not be added to the admin-override precedence level. Only mutable profiles may be added above the editing profile.

The admin precedence level will always contain a mutable profile named local_default.

The system precedence level will always contain an immutable profile named base.

The solaris.smf.modify authorization will allow users to modify precedence levels. If a user possesses the authorization to modify all of the properties which a profile contains values for, then she will be allowed to activate or deactivate that profile, also.

2.3.3. Profiles

SCF scope objects may contain named objects called profiles. Each will have a mutability flag, a nonpersistence flag, a stability attribute, an “origin” attribute, an optional predicate, and an inclusion list. Nonpersistent profiles will be discarded on system shutdown. Mutable profiles may contain service objects, and immutable profiles may contain manifest objects (described below). Profile inclusion lists will function like precedence levels.

Service and instance objects in mutable profiles may contain nonpersistent property groups. Such objects in immutable profiles may not.

Instance objects in profiles will have a “complete” flag.

Profiles may contain “negative entries” for services, instances, property groups, or properties. Through the naive interface they will mask corresponding entities in lower profiles making the entry appear as if nonexistent.

The solaris.smf.modify authorization will be required to create, destroy, or modify profile objects.

The solaris.smf.modify authorization also allows users to modify the contents of profiles. smf_security(5) also explains that additional authorizations can be specified which grant rights to modify portions of service configuration. These rights will be extended to inactive profiles, though the list of authorizations which may be used will always be based on the active values of the respective properties. For example, if a user possesses an authorization which allows him to manipulate the general/enabled property of a service, then she will be able to modify that property for that service in any profile.

Manifest Objects

Immutable profiles contain manifest objects. Manifest objects have names and hash attributes, and contain service objects. Manifest objects can be replaced, but not modified.

2.3.4. Backwards Compatibility

libscf Interface

A profile will be called “active” if it is referenced by a precedence level and it either has no effective predicate or its effective predicate evaluates to true, or if it is included by an active profile and it either has no effective predicate or its effective predicate evaluates to true.

Profiles may contain “negative entries” for service objects, instance objects, property group objects, or properties.

The scf_scope_get_service() and scf_iter_scope_services() functions will return service objects in active profiles which are not overridden by negative entries. The scf_scope_add_service() function will create a service object in the local profile. The scf_service_delete() function will create a negative service object in the local profile.

The scf_service_get_instance() and scf_iter_service_instances() functions will return instance objects in active profiles which are not overridden by negative entries, unless none of those objects assert the complete flag. The scf_service_add_instance() function will create an instance object in the local profile, with its complete flag set. If necessary an appropriate service object will be created in the local profile automatically. The scf_instance_delete() function will create a negative instance object in the local profile.

The scf_service_get_pg(), scf_iter_service_pgs(), scf_instance_get_pg(), and scf_iter_instance_pgs() functions will return property group objects which correspond to property group objects in the service or instance objects of active profiles. The attributes of the returned property groups will be those of the corresponding property group in the highest profile. The properties visible in property group objects will be the union of the properties in the corresponding profile property groups. The types and values of each property will be those of the property in the highest profile. Negative entries for property groups or properties will override profiles as expected.

Attempts to modify properties contained in profiles in the system-override precedence level will fail. Successful modifications to a property group will be recorded in the editing profile. If necessary, appropriate service and instance objects will be created automatically. (The complete flag of such instance objects will not be set.) Modifications to properties contained in profiles in admin-override but above editing (e.g., milestone) will result in those properties being deleted from the interfering profiles, and the changes being placed in editing.

When a snapshot is taken, only properties from local down will be copied. The running snapshot will be automatically retaken every time a profile from local down is changed. (Essentially, running will be a view of the properties without the profiles above local.)

A project-private operation will be added to flush the contents of the editing profile into local for a given service instance. svc.startd will use it to implement the refresh operation.

A project-private interface will be enhanced to emit notifications when the running snapshot changes for a service.

High-level Interfaces

svccfg delete will remove the manifest object from the repository, but leave in place any customizations for this service. Removal of the manifest object will make the service appear to be nonexistent from the naive interface. SMF will re-import the manifest when it has been changed and the customizations will reappear for this service. Third party software packages which upgrade by doing pkgrm, containing svccfg delete, followed by pkgadd will continue to work since the default behavior will leave customizations in place.

svccfg delete with a new option will remove the manifest object plus any customizations for this service from the local and editing profiles. SMF will re-import the manifest when it has been changed, but any administrative customizations made via the naive interface on the previous instantiation of this service will have been lost. This will be used by service developers who may be debugging new manifests and don't wish to leave lingering customizations in the repository after service removal.

svccfg delete is not viewed as a customization for the service so the deletion of a service will only be maintained until a new manifest appears preserving backward compatibility with existing behavior. The administrator will need to continue to disable a service to keep it turned off across an upgrade where a new manifest may be delivered.

svccfg revert will determine the differences between the current properties of a service and the target snapshot and will modify the editing profile to implement those differences. We considered putting the changes into a separate profile so that the user could easily undo the revert or see his pre-revert customizations, but this was deemed too complex and not strictly necessary.

svccfg apply will retain the ability to import profile files in today's format. The changes will be made to the local profile.

Incompatibilities

• Currently svcadm isn't permanent — it can be overridden by another tool (like NWAM). We are planning on making svcadm-initiated change permanent, which means tools like NWAM won't override them. We think the current behavior is a bug, so the change is justified. Measures may need to be taken to ensure users aren't surprised.

• If a naive client attempts to modify a property who's current value comes from a profile which is active in the system-override precedence level, it will fail. Note that denying property modification for reasons other than access control is new behavior. Since we anticipate it only being used for the publication of runtime information, primarily from svc.startd to the SMF tools, we think this is justified.

• Today importation of a service manifest which declares a service which doesn't exist as a dependent will result in the dependent service being created with only dependency configuration. The completeness flag will result in that configuration existing in a profile, but not visible as a complete service. We see no reason for programs to depend on the current behavior, so we don't anticipate this change causing any problems.

2.3.5. Administrative Features

Several administrative features will be supplied to assist the administrator in enhanced profile management. The features include:

To facilitate ease-of-use, libscf will provide functionality for administrative tools to determine when configuration customizations are overridden by a higher profile, so that the user can be warned.

Profile Verification

Users will want to determine when the reader-visible configuration differs from a particular profile. In particular, users will want to audit configuration against the secure by default (SBD) configuration values.

Simply checking to see whether that profile currently exists isn't sufficient because higher precedence profiles may arbitrarily override the configuration. Therefore we will offer an interface to administrators to compare the properties in a given profile against the currently-visible versions. The given profile does not need to be known by the system. In fact, it could be hand-crafted profile of service properties that the administrator wants to verify.

For example, the interface might be the verifyprof subcommand of the svccfg command. svccfg verifyprof profile would compare each of the properties of profile with the currently-visible versions, and report any differences.

Since any package will be allowed to deliver property settings into well-known system profiles, the administrator should understand the settings in the well-known profile before using it for verification. If the administrator doesn't check the profile used for verification, this could lead to the system being checked against different profile settings than expected. See open issue “Using well-known profiles for verification.”

This functionality will also be exposed for developers to employ in SCF-modifying tools. libscf(3LIB) will probably contain the interfaces.

2.3.6. Committed Profiles

Solaris will commit to some profiles for administrators to use to realize high-level characteristics of the system.

local_services

Enables services which offer service to local users (do not listen to the network) and would have been enabled by default on Solaris 10. Also keeps those services from accepting external network connections, if necessary.

secure_net_services

Enables services which listen to the network in a secure fashion, as determined by the Secure By Default team.

net_services

Enables the services which were enabled by default on Solaris 10 but are not enabled by local_services or secure_net_services. Also configures services to accept external network connections.

generic_limited_net

Contains property values to undo the effect of net_services, and includes secure_net_services. Provided for use with svccfg verifyprof.

generic_open

Includes net_services and secure_net_services, in that order. Provided for use with svccfg verifyprof.

Some profiles will be delivered for interim use by the installer.

ns_nis

Enables services required for a system to be a NIS client.

ns_nisplus

Enables services required for a system to be a NIS+ client.

ns_ldap

Enables services required for a system to be an LDAP client.

ns_dns

Enables services required for a system to be a DNS client.

We expect these to be replaced by the Duckwater project, so they will have lower stability.

2.3.7. Install

Today installation of a package containing an SMF service onto a live system automatically imports the service manifest into SCF. Delivered profile modifications will be handled similarly, so modifications to profiles which are active will take effect immediately. This could result in a service starting immediately, which doesn't happen today. Therefore this project will modify service installation to disable temporarily services created during package installation. (Maybe we shouldn't. See open issue “Temporary disable of new services.”)

Installation of a package containing an SMF service onto an unbooted root filesystem (pkgadd -R) today simply places the service manifest into a well-known directory. The manifest should be imported by the manifest-import service when the system is booted. Change manifests for enhanced profiles will be treated similarly.

As so far proposed, only system services would be enabled after a system was initially installed. Users expect more services to run by default, however. Therefore initial installation will automatically activate the local_services and the secure_net_services profiles. That should replicate the effect of applying the generic_limited_net.xml profile, as is the default today. Since the set of applications which should run is an administrative preference, these profiles will be activated to the admin precedence level, despite being placed there by the system. (Perhaps they should be activated to system. See open issue “local_services.”) In addition, the installer may direct other profiles to be activated. For example, the installer chooses which name service should be used, so the corresponding name service profile (ns_*) should be activated. The Duckwater project might change this.

Once NWAM begins using conditions, however, secure_net_services should not be activated unconditionally because without a network the services it enables would waste resources. Instead it should be included in the administrator's NWAM profiles, which usually should only be active in the presence of a network.

Backwards compatibility requires a few extra pieces of functionality. Administrators can modify the /var/svc/profile/generic.xml symlink to point to generic_open.xml between installation and first boot to direct the system to apply the generic_open.xml profile instead of the generic_limited_net.xml profile. To preserve this behavior (despite the absence of generic_limited_net.xml and generic_open.xml), manifest-import will examine the link and activate the net_services profile if it has been so modified. Administrators can also place a profile file at /var/svc/profile/site.xml; initial install will apply its properties to the local profile.

This project will also deliver an interface which administrators can use to deliver profiles into an unbooted system, and to specify how they should be activated on boot. This should be particularly useful for JumpStart users.

If more services are delivered with service manifests which specify that the service should be enabled on creation, as we will recommend, then svc.startd must be modified to cope. Today svc.startd starts enabled services as soon as their dependencies are met. But since services can install dependencies on other services (through dependent service manifest elements), and the manifest-import service does not import services in dependency order, this can lead to services being started before they should according to the final configuration. Today this is worked around by creating services disabled and enabling them in a profile which is applied at the end of manifest-import. Instead this project will enhance svc.startd to await manifest-import's completion before evaluating services for start.

2.3.8. Upgrade

When a service is upgraded from a version which does not participate in SMF to one that does, it should be enabled only if the old service would have run. Furthermore, if the new version stores any configuration in SCF, any customizations from the old configuration store should be migrated. SCF will designate a directory for the postinstall scripts of packages which deliver such services to place change manifests which encode configuration customizations. On boot these change manifests will be applied to the local profile and will be removed or renamed (to prevent subsequent application).

During system upgrade from Solaris with SMF to Solaris with enhanced profiles, SCF must populate repository profiles so that the same property values are visible through the naive interface, but with customizations separated from defaults. Reasonable inferences can be made by comparing the last-import snapshot, the running snapshot, and the current property values of each service. Administrators will be advised to verify SCF's customization determinations by examining the contents of the local and editing profiles after an upgrade across the transition.

If a new version of a service interprets configuration stored in SCF differently, then it is up to the service developer to migrate any user customizations. (The new defaults should be instituted automatically, as previously discussed.) The enhanced profiles model should make this much easier, since customizations can be located in mutable profiles. SMF currently doesn't offer a formal mechanism for doing this, so it must be done under the guise of other work, such as the start method. We urge the SMF team to introduce a mechanism. (See future work items “Service-specific Configuration Upgrade,” “Configuration versions.”)

Customization migration should also be able to include modifying the admin precedence level which includes activating profiles, deactivating profiles, renaming profiles, changing the predicates on profiles, making profiles include or not include other profiles. Therefore SCF should provide a way for developers to take a snapshot of not only the contents of mutable profiles but also the admin precedence level, so that the administrator will have the old configuration, if needed, for whatever reason.

2.3.9. Developer Guidelines

This project will change SMF guidance for developers. We'll cover the changes in five categories of developer deliverables and developer-delivered functionality: metadata, service implementation, administration, upgrade, and patching.

Metadata

If a developer's service should contribute to a system-wide profile, then he should deliver an appropriate change manifest into that profile. Such change manifests should not deliver properties for services other than those delivered by the author, except implicitly via the dependent element.

For services which are important to the system, developers should write their manifests to enable the services on creation. This will require an amendment to the SMF ARC policy.

For configuration which isn't pre-determinable, like configuration which should reflect system hardware, SMF does not provide metadata facilities. Instead the developer must create the configuration at runtime. This should be done in a new nonpersistent profile, which should be activated at the system precedence level. If the profile is to be unconditionally active, then there should be no reason for it to overlap the default configuration delivered in base. Thus, the order relative to base is irrelevant.

Service implementation

Services should continue to use existing interfaces to read configuration.

Since the enhanced profiles interfaces will allow clients to change properties on different services atomically, developers may wish to coordinate configuration changes between services. In effect, it would be bad for one service's process(es) to accept changes before the other's. Atomic changes make this possible, but two pieces of functionality would still be required. First, the service which should wait needs to be able to determine whether the other service's configuration has changed, otherwise there is no reason to wait. Second, the service must somehow detect when the other service has finished reading and applying the change. Currently, this must be service-specific. (We could imagine adding functionality to SMF to allow services to report when configuration has been read, which would allow a synchronous mode for svcadm refresh, but we will leave that for a future project.) When that is done, the first service may proceed to apply the configuration changes.

The repository could aid developers in the first part by allowing services to detect when the configuration of another service has changed, say by publishing signatures or timestamps for current configuration. But if the configuration is limited, it could also be done by storing and comparing, though this would require the first service to understand the second service's configuration. That would not be maintainable, but it means that the repository support isn't necessary for the initial delivery of this project. (See future work item “Configuration Signatures.”)

Services which disable themselves when they determine they shouldn't run should instead employ conditional configuration. The service should be delivered disabled and the scenario in which it is permissible for the service to run should be represented by a condition. The developer should deliver a profile which enables the service, and that profile should have a predicate consisting of the condition. If the service counts as “important to the system”, then that profile should be delivered into the system precedence level. Otherwise, if it fits the definition for one of the system-wide profiles, then the developer should arrange for his profile to be included in that profile. Otherwise, the profile should be delivered inactive so that the administrator can activate it at his discretion.

Conditional enablement won't prevent the service from being started, however. The administrator can still override the policy and enable the service despite the condition. If the service cannot function, it should fail into the maintenance state, to signal the error to the administrator.

We think it's likely that developers will want to create conditions based on the presence or absence of hardware devices. If the mechanism to notify the repository of condition changes is by libscf(3LIB) interfaces (as proposed below), then this will require the developer to write a daemon which detects when the device is present or absent, and use the libscf interface. If hald can detect the device, it's likely that the daemon would just subscribe to hald's events and forward them to the libscf(3LIB) interface. We could greatly simplify this for the developer by allowing him to just declare that the condition should be tied to a hald device, and either having a single daemon which handles all such conditions, or building this functionality directly into hald or the repository. In fact, if hald device designators are sufficiently constrained, the repository could interpret a class of condition names as being automatically tied to certain hald events. The naming constraints and conventions for conditions should take this into consideration. (See future work item “Device conditions.”)

Developers who want their service to disseminate status information through SCF should do so by setting properties in the generic_status profile.

It seems reasonable that developers will want to store persistent state data in SCF. Assuming that such data should not be overridden by the administrator, it should be kept in the system-override precedence level. To that end we could define a generic_state profile there. We know of no consumers at this time, however, so we'll leave it to define in the future. (See future work item “generic_state.”)

Administration

When a developer chooses to store configuration in SCF, he should consider how administrators will customize it. For no extra development work, the administrator can use SMF's generic tools, svcadm & svccfg, to make customizations. This is usually a rather low-level interface, though, and invalid configuration won't be detected until it is read by the service implementation.

If there are only a few legitimate customization possibilities, a developer could deliver service-specific profiles for the administrator to select customizations. In this case the developer might direct the administrator to consider the profiles as the public interface, and the properties as private, so generic tools should not be used to manipulate them. We urge the SMF team to permit developers to formalize this, so that generic tools can warn administrators who attempt to manipulate such properties.

If configuration properties need to conform to nontrivial relationships in order to be valid, or the consequences of invalid configuration are severe, or the developer wishes to provide a higher-level configuration interface, then the developer should deliver a tool for administrative use. Such tools may use today's naive libscf interface, but it doesn't permit expression of temporal attributes (e.g., tied to a condition) and it retains incomplete configuration changes, which complicates error recovery.

Therefore we encourage developers of such tools to utilize the profile-aware interfaces. By default tools should make changes in the local_default profile, but they should also allow the user to specify a profile and attempt to make changes in it. Either way the tool should use libscf functionality to determine whether the new configuration is being overridden, and warn the user if so.

Administrators may require more elaborate profile-related functionality from administrative tools, such as moving settings between profiles and distilling common configuration from multiple profiles into a single included profile. If this functionality becomes common, it should be integrated into libscf.

Since we will allow developers to “create” instances in profiles (via the completeness flag), developers will be tempted to do so in conditionally active profiles. For example, instances created by ksslcfg might only make sense in certain NWAM network environments. The restarter responsible for those instances should be taken into consideration, however. svc.startd, for example, reads stop methods on-demand, so if a service instance disappears, it will no longer have access to its stop method. Arguably, svc.startd should cache the stop methods of services, but while it doesn't we recommend that developers “create” such instances in a permanently active profile, such as local_default, and simply enable them in the conditionally active profile.

Upgrade

SMF will automatically upgrade default configuration when upgraded packages deliver new service manifests and change manifests.

To migrate configuration customizations into SCF, developers should translate the customizations into change manifests in a designated directory on the filesystem. Upgrade of customizations within SCF should only be necessary when the meaning or encoding of configuration changes.

Customizations created by the administrator, usually in mutable profiles, should be rewritten to preserve intent. This cannot be done in the packaging system's designated hook, the postinstall script, since SCF forbids reading or writing the repository of a different version of Solaris. It can either be done as part of the service implementation (e.g., the start method), or through the /var/svc/profile/upgrade script, which the SMF team has deemed private and thus requires an ARC contract to use. This extra work and the absence of formal SCF upgrade support makes developer-delivered customization profiles more attractive.

Note that configuration customizations are not limited to the contents of profiles. Entries of the admin and admin-override precedence levels and the inclusion lists of administrator-created profiles are also customizations, and may need to be migrated on upgrade.

Patching

Some developers may want to modify configuration temporarily during patching. Specifically, developers may want to disable a service while it is patched. We believe it would be better for the developer to fail the patch operation if the service is running, so that the administrator is aware of the ramifications and they happen at his discretion.

2.3.10. Structural Diagram

The diagram below illustrates the relationships between the elements of this model. The dashed lines indicate objects synthesized for backwards compatability.

2.4. Scope

These items are beyond the scope of this project and won't be addressed:

• Problems with the service/instance split.

• User-initiated snapshots.

• running snapshot stability during start.

2.5. Open Issues

This section describes the current unresolved issues concerning the design of Enhanced Profiles:

Configuration Semantics

Does Enhanced Profiles need to make special provisions to accommodate different classes of configuration data such as immediate configuration data (general/enabled) and only-read-on-start (start/exec) configuration data? The former requires different refresh semantics and the latter might ideally require treating service stop as a configuration-change boundary. Would teaching the tools about this (e.g., "templates") suffice?

Condition Strings

Some sets of mutually exclusive conditions would be better represented by a string which can take on one of a set of values. Instead of a separate boolean condition for each platform, core Solaris could deliver the platform variable, which may be set to an identifier. Profile predicates could then test the variable against a particular identifier.

Before this can be instituted, changing type during an upgrade should be understood. Suppose NWAM represented the network environment with a string, but later was capable of recognizing multiple network environments simultaneously. The network environment variable could take on concatenations of network environment names, or a condition could be created for each network environment, and the set would not be mutually exclusive. In the latter case, predicates on the user's profiles would have to be migrated from the string comparisons to boolean condition references. If this should in fact be done during upgrade, then it should not be done until SCF allows developers to take snapshots of profile predicates prior to condition deliverer's migration. (See future work item “Profile snapshots.”)

Sophisticated Condition Constraints

It seems that there are condition relationships which mutual exclusion groups don't work well for. For example, “platform_t2000” implies “sun4v”, but expressing this with mutual exclusion groups would be awkward and not scalable. Should we allow condition developers to specify expressions which are always true or are never true? If not, user interfaces might make recommendations based on impossible scenarios.

Device Instances

It seems that creating instances for devices is a trend. Is there anything Enhanced Profiles should do now to facilitate settling that into the SMF framework? Should hald be employed?

local_services

Should local_services be activated in the system precedence level instead of the admin precedence level? That should purify the admin and admin-override levels as customization-only, and would make it easier to change the default profile list, but some users could rightfully demand an anti-local_services profile.

Modification Interfaces

Should the current modification interfaces be obsoleted in preference for the new profile modification interfaces?

Device Conditions

Part of the condition namespace should be reserved for conditions tied to devices and their attributes. How should the namespace be structured in anticipation?

Profile Types

The special profiles are a kind of profile type. Should profile types be explicit?

Removal of service package

When a package is removed that contains a change manifest for a service, then the change manifest will be removed. What will happen to administrator created profiles that include the removed change manifest? What happens to any properties associated with the service or its instances that are in the local profile?

Temporary disable of new services

If an administrator installs a package which contains an SMF service and the installation mechanics temporarily disable it, then to start it before the next boot the administrator will be inclined to use svcadm enable. But that svcadm enable will be recorded in the local profile as a customization. What if the administrator wanted the service enabled because an active profile specified that it should be enabled? Will it be ok to make administrators learn a new command for that case? Or would it be better to just let the service start when it is installed?

Using well-known profiles for verification

Should the administrator trust a well-known profile in order to verify service property settings on the system? Any package is allowed to deliver property settings into a file in the well-known profile directory. By convention, the delivered file should only contain property settings for services delivered by that package. One failure situation is when the profile verification fails due to the system not matching the property value setting for a service not relevant to the administrator. This leads to the administrator spending time and effort to determine why the profile verification failed. A worse situation happens if the delivered file deviously overrides a property set in another file in that profile directory. This could lead the administrator to believe that the system is running with specific property values which aren't true.

Permanent profiles to override conditional profiles

Should permanent profiles be allowed to be placed above conditional profiles at the same precedence level? The only reason to override permanently a condition profile is when a user is not authorized to deactivate the condition profile.

Runlevels, NWAM, and the service level profiles

I suspect there is some deep connection between runlevels, NWAM, and the service level profiles. Is the difference between multi-user and multi-user-server more than net_services & secure_net_services? Is the difference between single-user and multi-user more than local_services?

2.6. Future Work

Listed below are ideas for future SMF work that have come up during the design of Enhanced Profiles:

Service-specific Configuration Upgrade

Migrating configuration customizations for a service is difficult because the hook offered by the packaging system (the postinstall script) can be executed for an unbooted root filesystem, for which SCF forbids reading the repository. Instead, developers must currently handle the migrations either as part of their service, which requires the service to have sufficient privilege to write modify configuration (as opposed to simply reading it), or with the upgrade script, which is private to SMF. It seems that SMF should offer service authors a hook which can be executed with higher privilege when the configuration needs to be upgraded.

Profile snapshots

Profile snapshots will be taken when a service upgrades a mutable profile. This will allow the administrator to revert back to a previously working profile setup. It will also make it easier for administrators to work around bugs in the migration.

If upgrade will be allowed to manipulate precedence levels, then they should be snapshot-able, too.

Configuration versions

A service will have a configuration version stored in the repository. When a service is started it will check the configuration version that is stored in the repository. If the configuration version doesn't match the version that the software is currently operating at, then the service may need to upgrade its properties, including the configuration version associated with its data in the repository. This is useful when a service is upgraded.

svccfg revert should be reconsidered in light of this feature.

Configuration signatures

A configuration signature will be added to each service in the repository. When configuration data is changed in the repository then the