CPU Power Management
OpenSolaris provides support for Dynamic Frequency and Voltage Scaling (DVFS) across a range of Intel and AMD based processors. DVFS provides a mechanism
enabling a processor to operate across a range of clock frequencies and
voltages, allow one to trade off performance vs. power consumption. Intel's "Enhanced Speedstep Technology", and AMD's "PowerNOW!" are examples of
DVFS. On x86 architectures, DVFS features are exposed to the OpenSolaris kernel through ACPI Performance States (P-States), where a P-State is
an abstraction for a power/performance state. Changing P-states will typically
cause the processor to change it's operating voltage and frequency, resulting
in a corresponding change in performance, and power consumed.
The ACPI standard also defines Core States (C-States), where a C-State is an
abstraction for a state that a processor may enter while idle. Entering the C-State will typically cause the processor to suspend instruction execution.
Because the processor doesn't have to do any work, it may off-line various
micro-architectural components facilitating an even lower dropping of voltage
than would be possible in active operation. Entering deeper C-States results
in lower power consumption, at the cost of an increased latency to recover to
active operation.
This page describes the Tesla team's project work to collectively provide OpenSolaris with a next-generation CPU Power management architecture, fully event-driven, predictive, adaptive, and integrated with the kernel's thread scheduler/dispatcher subsystems.
Projects
CPU power management as it it implemented today is relatively isolated from
the rest of the system. As such, it is forced to periodically poll to measure
the utilization of the system's CPU resources. When CPU utilization drops to
a sufficiently low level, the power state (P-State) of the CPU is dropped.
Likewise, when a CPU's utilization increases, the P-State is raised to higher
performing state. The present day architecture suffers from a few shortcomings:
- Polling is a poor thing for the power management subsystem to
have to do. First, the question arises...how often to poll? There is a trade off
that arises around polling more often to improve responsiveness to changes in
utilization, vs. polling less often to minimize overhead. The current
architecture polls relatively infrequently...which means that there are
non-trivial durations of time where a thread may run on a clocked-down CPU before the PM subsystem notices that utilization has increased (and the CPU should be clocked up)...or where the CPU has become idle, but remains clocked up until the PM subsystem notices it should be clocked down. Polling is also
inefficient, because it means that even on an otherwise quiescent system, the
power management implementation still needs to wake up (bringing at least some
resources into a higher power consuming state) to check to see if the system
is still idle.
- Power Management decoupled from Resource Management - The thread
dispatcher is the kernel subsystem responsible for deciding where
(on which CPUs) threads should be scheduled to run. At present, it has no
notion of CPU power/performance states. At the same time, the CPU power management subsystem is polling looking for idle CPU resources to power manage. Having these two subsystems decoupled leads to situations where the two subsystems can undermine each other's efforts, leading to poorer performance as threads are inadvertantly run on clocked down CPUs, or where utilization across the system remains light, but is distributed across the system to the point where nothing is quiescent enough to be power managed.
This project extends the kernel's existing topology aware scheduling facility
to bring "power domain" awareness to the dispatcher. With this awareness
in place, the dispatcher can implement coalescence dispatching policy to consolidate utilization onto a smaller subset of CPU domains, freeing up other
domains to be power managed. In addition to being domain aware, the dispatcher
will also tend to prefer to utilize domains already running at higher
power/performance states...this will increase the duration and extent to which domains can remain quiescent, improving the kernel's ability to take advantage
of features like deep C-states. Because the dispatcher will track power domain
utilization along the way, it can drive active domain state changes in an event
driven fashion, eliminating the need for the CPUPM subsystem to poll.
Status
PAD and Deep C-states support integrated into Nevada build 110 on Wednesday Feb 25th, 2009. For more information, please see the Flag Day and Heads Up announcement.
Bugs/RFE CR numbers
Integration:
-
6567156 bring CPU power awareness to the dispatcher
Documents
Power Aware Dispatcher Source Repository
- pad-gate: This repository is closed, as the project has integrated. Please see the current ON source base
Modern x86 processors support several different idle states for power conservation. ACPI defines these as C-states. Solaris as of onnv_102
supports only ACPI C1 via the HLT (halt) and MONITOR/MWAIT instructions.
Deeper ACPI C-states C2 and C3 can conserve more power, but they can take
longer to enter and resume. The ACPI specification allows CPU internal
clock state to halt during C2, and caches may loose state in ACPI C3.
Operating system support is required because of possible CPU state loss
in C2 and C3 and because of the additional idle wakeup latency.
There is currently OpenSolaris Deep C-state work ongoing in several areas:
- cpudrv The existing cpudrv is being modified to support Deep C-states. The cpudrv is being modified to:
- detect processors which support deep C-states
- query ACPI properties
- implement idle loops to enter deep C-states via ACPI methods.
- kernel General kernel work to support Deep C-states
- support different CPU wakeup mechanisms for CPUs in different idle states.
- Read timers such as the local APIC and expire times such as for the top
cyclic on a CPU's cyclic heap
- Scheduler improvements to choose CPUs based on idle states.
- support Real Time (RT) thread scheduling time requirements on CPUs with
variable wakeup latencies
- HPET Solaris uses the local APIC timer to generate interrupts for
the Cyclic Backend (CBE). The lAPIC timer in a CPU may stop counting and
will not generate interrupts while the processor is in ACPI states C2 and C3.
Ongoing work is being done to use the High Precision Event Timer (HPET)
as a proxy for stalled lAPIC timers. The HPET is located on the chipset
isolated from CPU C-State power side effects. CPU must schedule their
next CBE interrupt on the HPET when they enter a deep C-state.
Status
Deep C-states support integrated (along with PAD) into Nevada build 110 on Wednesday Feb 25th, 2009. For more information, please see the Flag Day and Heads Up announcement.
Bugs/RFE CR numbers
C-State work has merged with Power Aware Dispatcher work. Please see pad-gate above.
HPET and C-State work was developed in separate gates to maintain quality
of other gates. These gates are no longer active.
Mercurial Repositories
Please read
these instructions on how to use Mercurial repositories. For help with using Mercurial, or the ON tools, you can also:
To make a (debug) kernel (using pad-gate as an example workspace, and opensolaris.sh as the environment file)
$ cd pad-gate
$ /opt/onbld/bin/bldenv -d /opt/onbld/bin/opensolaris.sh
$ cd usr/src/tools
$ dmake install
$ cd $CODEMGR_WS/usr/src/uts
$ dmake install
To create a kernel tarball to install (x86)...
$ /opt/onbld/bin/Install -G my_pad-gate_kernel -k i86pc
To build BFU archives, you need to get (and extract) the "closed bins" tarball(s) into your workspace. See above for current pointers (you must use versions appropriate for the build of onnv against which your repo is synced).
$ cd pad-gate
$ tar xf on-closed-bins.i386.tar
$ /opt/onbld/bin/nightly /opt/onbld/bin/opensolaris.sh
See the
OpenSolaris Developer's Reference for details on how to use
kernel tarballs generated by Install(1).
|