OpenSolaris

Energy Star Guidelines for Kernel Programmers

When deploying a new kernel thread in Solaris, please be aware that it is necessary to follow a protocol which allows the thread to be quiesced in a safe manner when cpr is suspending the system. The protocol was developed so that kernel threads are given notice that the system is going to be shut down. Cpr requires that ALL threads in the system be quiesced so that kernel memory doesn't change as the pages are being written to the statefile.

The operation of fsflush() is a good illustration of how the callback protocol works:

#include <sys/callb.h>

In the initialization of the fsflush thread, before the main service loop is entered, it invokes the CALLB_CPR_INIT macro. This initializes a data structure and enters the thread in the callback table (see callb.c):

For example,

• CALLB_CPR_INIT(&cprinfo, &fsflush_lock, callb_generic_cpr, "fsflush");
• cprinfo is declared on a per-thread basis as "callb_cpr_t cprinfo". It may be a local or static, but must exist until CALLB_CPR_EXIT is called. It contains a set of flags, a condition variable for the thread to block on and another cv for cpr to block on, but the contents are really opaque; access to the structure is done exclusively by the generic callback code and the macros.
• fsflush_lock is a mutex which protects the above data structure. Often, a kernel thread can overload an existing mutex for use by this callback protocol. Fsflush() uses the same mutex that it uses in conjunction with the cv which blocks until the clock thread signals it to run again.
• callb_generic_cpr is a function pointer and should always be hard-coded as the third argment of the CALLB_CPR_INIT macro call. It gets called via the callback mechanism when cpr needs to suspend or resume the kernel thread.
• "fsflush" is a string which identifies this particular thread in the callback table. It should be descriptive, short, and unique. Max length is 23 characters.
Note: If CALLB_CPR_INIT() is invoked when cpr has already begun to suspend the system, the kernel thread will block until the system is resumed by cpr.

In the service loop of the kernel thread:

When the thread is at a point in its processing where it can safely be suspended, it calls the CALLB_CPR_SAFE_BEGIN macro. The best location for this call is when the thread has just finished doing some work and has yet to begin a new task.

For example,

• CALLB_CPR_SAFE_BEGIN(&cprinfo);
• cprinfo must be the same as the first argument to the CALLB_CPR_INIT macro above. Again, this is the structure which is accessed both by the thread and cpr to allow the callback mechanism to work. Before invoking this call, the thread must own the lock declared as the second argument to the CALLB_CPR_INIT invocation.
When the thread is about to begin some work, during which it must not be interrupted, it should declare that it is no longer safe to be suspended. This is done by invoking the CALLB_CPR_SAFE_END macro:

For example,

• CALLB_CPR_SAFE_END(&cprinfo, &fsflush_lock);
• cprinfo must be the same as the first argument to the CALLB_CPR_INIT macro above.
• fsflush_lock refers to a lock which will be used in a cv_wait() call. The cv_wait() is executed if the flags in the data structure indicate that cpr still requires the thread to wait. In almost all cases, this argument can be the same as the the second argument of the CALLB_CPR_INIT invocation. Notice that fsflush() uses the same lock.
Fsflush() represents the simple case, because it's a kernel thread which doesn't exit. If your new kernel thread can exit, then it requires an additional macro invocation. A good example of a thread like this is nfs_async_start(). The use of the three macros described above will be the same for a thread which can exit. In addition, before the thread exits, it must invoke the CALLB_CPR_EXIT macro. This call insures that the thread's entry will be removed from the callback table.

For example,

• CALLB_CPR_EXIT(&cprinfo);
• cprinfo must be the same as the first argument to the CALLB_CPR_INIT macro.

The lock which was passed as the first argument of the CALLB_CPR_INIT macro invocation must be held when CALLB_CPR_EXIT is invoked. *** NOTE *** CALLB_CPR_EXIT does a mutex_exit() on the lock which is passed in as an argument. If an existing kernel thread is being modified to comply with the cpr protocol, and if the thread shares an existing lock to also protect the cprinfo structure, BE CAREFUL NOT TO RELEASE THE LOCK TWICE! If there is an existing mutex_exit for the shared lock, it may need to be removed.

Note: The macro definitions in callb.h contain some references to an obsolete flag. This will be cleaned up shortly. Also, comments will be added to thread.h and thread.c explaining the need to conform to the protocol described here.

---

If you have comments or questions about this, please contact estar-dev at sun dot com.