Recently at Stackdriver, we wanted to understand the metric of “CPU steal”. We knew that it was a relatively recent addition to the Unix method of tracking CPU usage, and we knew that it was introduced with the growing use of virtualization. We wanted to understand the metric better to so that we could provide better insight into your cloud infrastructure.
CPU steal is a measure of the fraction of time that a machine is in a state of “involuntary wait.” It is time for which the kernel cannot otherwise account in one of the traditional classifications like user, system, or idle. It is time that went missing, from the perspective of the kernel.
CPU steal is primarily relevant in virtualized environments in which a single host (physical) machine runs multiple guest (virtual) machines. Each virtual machine receives of fraction of the physical CPU resources (and other physical resources). That is, the physical CPU is multiplexed across the multiple virtual CPUs used by the guest VMs. The operating system kernel detects when it has work available but does not have access to the CPU to perform that work.
In order to visualize the behavior of contending systems, we ran some controlled experiments. We provisioned a physical server from SoftLayer. The machine contained a Intel Xeon (Lynnfield) Quadcore and 8 GB of memory. It ran Citrix XenServer 6.1.0. We configured two virtual machines running CentOS 6.4. Each virtual machine is configured to receive equal shares of the CPU.
We ran both virtual machines and applied various synthetic load using the lookbusy load generator. We used lookbusy to simulate processing load only. In the series of experiment reported below, we held the load on one virtual machine constant while we varied the load on the other virtual machine in steps of 0%, 20%, 40%, 60%, 80%, and 100%. We recorded the CPU state as reported by the kernel in the virtual machines.
In the first experiment, the no load is applied to one of the machines. The other machine is able to access enough CPU resources to run the stepped load. At higher utilization, there is only a slight bit of CPU steal detected due to the small amount of CPU needed to service the otherwise nearly idle other machine.
In the second experiment, a constant load of 25% is applied to one of the machines. As the load on the other machine grows, each machine begins to show higher amounts of CPU steal. Note that the machine with the stepped load is not able to consume more than 75% of the CPU because the other machine is using 25% from its fair share.
The pattern continues in the third experiment. Because the constant load is 50%, the machine with the stepped load is not able to use more than 50% of the CPU, despite the larger amount of work it is asked to perform. As the load approaches 50% for each machine, they alternate performing work and waiting on the other virtual machine to complete its time slice. Nearly no idle time is reported.
When a constant load of 75% is applied to one machine, it receives more than its fair share, as long as utilization of the other machine is low. As the other machine requires more CPU, both machines settle in to using 50% of the processor. When each machine is requesting more than 50% of the CPU, the machines again alternate between computing and waiting.
In the final experiment, one machine is given a constant load of 100% CPU utilization. The trends are similar to the previous experiment.
These experiments reveal several insights. First, when total utilization of the of the physical processor is low, guest virtual machines report very little CPU steal. Because the CPU is mostly idle, the virtual machines are able to access it almost always when they need it. The kernel measures CPU steal before the physical processor is fully utilized. Even though there is more than enough processor to satisfy all the VMs, sometimes the VMs must wait for access to the CPU. Finally, when the physical process is nearly fully utilized, the guests report effectively no idle time. The VMs alternate between processing (CPU user) and waiting (CPU steal).
Finally, it should be noted that these experiments were run in a very controlled setting on a stock Xen hypervisor. The behavior of VMs in public clouds could vary. It is known, for example, that AWS uses a Xen-based hypervisor. The community does not know, however, what modifications AWS may have made to the hypervisor and how they have it configured. While I expect that many of the insights from the experiments to apply elsewhere, behavior in other environments may differ.