US 20060136653 A1
The present invention is directed to making a guest operating system aware of the topology of the subset of host resources currently assigned to it. At virtual machine boot time a Static Resource Affinity Table (SRAT) will be used by the virtualizer to group guest physical memory and guest virtual processors into virtual nodes. Thereafter, in one embodiment, the host physical memory behind a virtual node can be changed by the virtualizer as necessary, and the virtualizer will provide physical processors appropriate for the virtual processors in that node.
1. A method for optimizing performance of an operating system executing on a computer system, said computer system having a dynamic virtual processor topology, said method comprising updating said operating system after startup with at least one update to reflect at least one change in said virtual processor topology.
2. The method of
3. The method of
4. The method of
5. The method of
6. The method of
7. The method of
8. A computer-readable medium comprising computer-readable instructions for optimizing performance of an operating system executing on a computer system, said computer system having a dynamic virtual processor topology, said computer-readable instructions comprising instructions for updating said operating system after startup with at least one update to reflect at least one change in said virtual processor topology.
9. The computer-readable medium of
10. The computer-readable medium of
11. The computer-readable medium of
12. The computer-readable medium of
13. The computer-readable medium of
14. The computer-readable medium of
15. A system for optimizing performance of an operating system executing on a computer system, said computer system having a dynamic virtual processor topology, said system comprising at least one subsystem for updating said operating system after startup with at least one update to reflect at least one change in said virtual processor topology.
16. The system of
17. The system of
18. The system of
19. The system of
20. The system of
The present invention generally relates to the field of virtual machines (also known as “processor virtualization”) and to operating systems that execute in virtual machine environments. More specifically, the present invention is directed to systems and methods for exposing the processor topology of a virtual machine to a guest operating system executing on a virtual machine wherein said topology is dynamic based on allocations of host computer system processor and memory resources.
Computers include general purpose central processing units (CPUs) or “processors” that are designed to execute a specific set of system instructions. A group of processors that have similar architecture or design specifications may be considered to be members of the same processor family. Examples of current processor families include the Motorola 680×0 processor family, manufactured by Motorola, Inc. of Phoenix, Ariz.; the Intel 80×86 processor family, manufactured by Intel Corporation of Sunnyvale, Calif.; and the PowerPC processor family, which is manufactured by Motorola, Inc. and used in computers manufactured by Apple Computer, Inc. of Cupertino, Calif. Although a group of processors may be in the same family because of their similar architecture and design considerations, processors may vary widely within a family according to their clock speed and other performance parameters.
Each family of microprocessors executes instructions that are unique to the processor family. The collective set of instructions that a processor or family of processors can execute is known as the processor's instruction set. As an example, the instruction set used by the Intel 80×86 processor family is incompatible with the instruction set used by the PowerPC processor family. The Intel 80×86 instruction set is based on the Complex Instruction Set Computer (CISC) format. The Motorola PowerPC instruction set is based on the Reduced Instruction Set Computer (RISC) format. CISC processors use a large number of instructions, some of which can perform rather complicated functions, but which require generally many clock cycles to execute. RISC processors use a smaller number of available instructions to perform a simpler set of functions that are executed at a much higher rate.
The uniqueness of the processor family among computer systems also typically results in incompatibility among the other elements of hardware architecture of the computer systems. A computer system manufactured with a processor from the Intel 80×86 processor family will have a hardware architecture that is different from the hardware architecture of a computer system manufactured with a processor from the PowerPC processor family. Because of the uniqueness of the processor instruction set and a computer system's hardware architecture, application software programs are typically written to run on a particular computer system running a particular operating system.
Computer manufacturers want to maximize their market share by having more rather than fewer applications run on the microprocessor family associated with the computer manufacturers' product line. To expand the number of operating systems and application programs that can run on a computer system, a field of technology has developed in which a given computer having one type of CPU, called a host, will include a virtualizer program that allows the host computer to emulate the instructions of an unrelated type of CPU, called a guest. Thus, the host computer will execute an application that will cause one or more host instructions to be called in response to a given guest instruction, and in this way the host computer can both run software designed for its own hardware architecture and software written for computers having an unrelated hardware architecture.
As a more specific example, a computer system manufactured by Apple Computer, for example, may run operating systems and program written for PC-based computer systems. It may also be possible to use virtualizer programs to execute concurrently on a single CPU multiple incompatible operating systems. In this latter arrangement, although each operating system is incompatible with the other, virtualizer programs can host each of the several operating systems and thereby allowing the otherwise incompatible operating systems to run concurrently on the same host computer system.
When a guest computer system is emulated on a host computer system, the guest computer system is said to be a “virtual machine” as the guest computer system only exists in the host computer system as a pure software representation of the operation of one specific hardware architecture. The terms virtualizer, emulator, direct-executor, virtual machine, and processor emulation are sometimes used interchangeably to denote the ability to mimic or emulate the hardware architecture of an entire computer system using one or several approaches known and appreciated by those of skill in the art. Moreover, all uses of the term “emulation” in any form is intended to convey this broad meaning and is not intended to distinguish between instruction execution concepts of emulation versus direct-execution of operating system instructions in the virtual machine. Thus, for example, the Virtual PC software created by Connectix Corporation of San Mateo, Calif. “emulates” (by instruction execution emulation and/or direct execution) an entire computer that includes an Intel 80×86 Pentium processor and various motherboard components and cards, and the operation of these components is “emulated” in the virtual machine that is being run on the host machine. A virtualizer program executing on the operating system software and hardware architecture of the host computer, such as a computer system having a PowerPC processor, mimics the operation of the entire guest computer system.
The virtualizer program acts as the interchange between the hardware architecture of the host machine and the instructions transmitted by the software (e.g., operating systems, applications, etc.) running within the emulated environment. This virtualizer program may be a host operating system (HOS), which is an operating system running directly on the physical computer hardware (and which may comprise a hypervisor, discussed in greater detailed later herein). Alternately, the emulated environment might also be a virtual machine monitor (VMM) which is a software layer that runs directly above the hardware, perhaps running side-by-side and working in conjunction with the host operating system, and which can virtualize all the resources of the host machine (as well as certain virtual resources) by exposing interfaces that are the same as the hardware the VMM is virtualizing. This virtualization enables the virtualizer (as well as the host computer system itself) to go unnoticed by operating system layers running above it.
To summarize, processor emulation enables a guest operating system to execute on a virtual machine created by a virtualizer running on a host computer system, said host computer system comprising both physical hardware and a host operating system.
Processor and Memory Topology
Modern operating system schedulers take into account the processor and memory topology of the machine to maximize performance. This is usually done at startup and, for an operating system executing on physical hardware, this is usually sufficient because the processor topology of physical hardware remains constant. The Windows Operating System (Windows XP, Windows 2003) and other operating systems typically determine the topology of the system at boot time in two ways: (a) by examining the memory and processor node topology information in the BIOS Static Resource Affinity Table (SRAT) and (b) by reading self-contained processor identification data (CPUID in x86/x64 processors) to determine specific Simultaneous Multithreading (SMT, a.k.a. hyperthreading) and multicore topologies.
As used herein, the term “processor topology” is broadly intended to refer to physical characteristics of the processor and associated memory that, if known by an operating system, could theoretically enable an operating system to better utilize the associated processor resources. Processor topology may include, but is not limited to, the following: static processor information such as SMT, multicore, and BIOS' SRAT data and/or information; static NUMA information such as processor, memory, and I/O resource arrangements; and any changes to the foregoing.
In a virtual machine environment, however, while the physical processor topology for the “hosting agent” (the host operating system, virtual machine monitor, and/or hypervisor) remains constant, the physical resources assigned to a virtualizer, and thus the virtual machine, may vary rapidly over time, making the topology assumptions made by the guest operating system running on the virtual machine inaccurate and hence inefficient.
While the dynamic nature of the topology can be mitigated by always using the same physical processor assignments for virtual processors or by limiting the assignments to a specific node, this would severely and negatively impact the virtualizer's ability to make optimal use of all host resources. Therefore, what is needed in the art is means for rectifying the inefficiency of a changing virtual topology without negatively impacting the virtualizers ability to make optimal use of all host resources.
Various embodiments of the present invention are directed to systems and methods for making a guest operating system aware of the topology of the subset of host resources currently assigned to it. For certain of these embodiments, at virtual machine boot time a Static Resource Affinity Table (SRAT) will be used by the virtualizer to group guest physical memory and guest virtual processors into virtual nodes. Thereafter the host physical memory behind a virtual node can be changed by the virtualizer as necessary, and the virtualizer will provide physical processors appropriate for the virtual processors in that node.
The foregoing summary, as well as the following detailed description of preferred embodiments, is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the invention, there is shown in the drawings exemplary constructions of the invention; however, the invention is not limited to the specific methods and instrumentalities disclosed. In the drawings:
The inventive subject matter is described with specificity to meet statutory requirements. However, the description itself is not intended to limit the scope of this patent. Rather, the inventor has contemplated that the claimed subject matter might also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the term “step” may be used herein to connote different elements of methods employed, the term should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.
Numerous embodiments of the present invention may execute on a computer.
As shown in
A number of program modules may be stored on the hard disk, magnetic disk 29, optical disk 31, ROM 24 or RAM 25, including an operating system 35, one or more application programs 36, other program modules 37 and program data 38. A user may enter commands and information into the personal computer 20 through input devices such as a keyboard 40 and pointing device 42. Other input devices (not shown) may include a microphone, joystick, game pad, satellite disk, scanner or the like. These and other input devices are often connected to the processing unit 21 through a serial port interface 46 that is coupled to the system bus, but may be connected by other interfaces, such as a parallel port, game port or universal serial bus (USB). A monitor 47 or other type of display device is also connected to the system bus 23 via an interface, such as a video adapter 48. In addition to the monitor 47, personal computers typically include other peripheral output devices (not shown), such as speakers and printers. The exemplary system of
The personal computer 20 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 49. The remote computer 49 may be another personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the personal computer 20, although only a memory storage device 50 has been illustrated in
When used in a LAN networking environment, the personal computer 20 is connected to the LAN 51 through a network interface or adapter 53. When used in a WAN networking environment, the personal computer 20 typically includes a modem 54 or other means for establishing communications over the wide area network 52, such as the Internet. The modem 54, which may be internal or external, is connected to the system bus 23 via the serial port interface 46. In a networked environment, program modules depicted relative to the personal computer 20, or portions thereof, may be stored in the remote memory storage device. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used. Moreover, while it is envisioned that numerous embodiments of the present invention are particularly well-suited for computerized systems, nothing in this document is intended to limit the invention to such embodiments.
From a conceptual perspective, computer systems generally comprise one or more layers of software running on a foundational layer of hardware. This layering is done for reasons of abstraction. By defining the interface for a given layer of software, that layer can be implemented differently by other layers above it. In a well-designed computer system, each layer only knows about (and only relies upon) the immediate layer beneath it. This allows a layer or a “stack” (multiple adjoining layers) to be replaced without negatively impacting the layers above said layer or stack. For example, software applications (upper layers) typically rely on lower levels of the operating system (lower layers) to write files to some form of permanent storage, and these applications do not need to understand the difference between writing data to a floppy disk, a hard drive, or a network folder. If this lower layer is replaced with new operating system components for writing files, the operation of the upper layer software applications remains unaffected.
The flexibility of layered software allows a virtual machine (VM) to present a virtual hardware layer that is in fact another software layer. In this way, a VM can create the illusion for the software layers above it that said software layers are running on their own private computer system, and thus VMs can allow multiple “guest systems” to run concurrently on a single “host system.” This level of abstraction is represented by the illustration of
Referring again to
In regard to
All of these variations for implementing the virtual machine are anticipated to form alternative embodiments of the present invention as described herein, and nothing herein should be interpreted as limiting the invention to any particular emulation embodiment. In addition, any reference to interaction between applications 116, 118, and 120 via VM A 108 and/or VM B 110 respectively (presumably in a hardware emulation scenario) should be interpreted to be in fact an interaction between the applications 116, 118, and 120 and the virtualizer that has created the virtualization. Likewise, any reference to interaction between applications VM A 108 and/or VM B 110 with the host operating system 104 and/or the computer hardware 102 (presumably to execute computer instructions directly or indirectly on the computer hardware 102) should be interpreted to be in fact an interaction between the virtualizer that has created the virtualization and the host operating system 104 and/or the computer hardware 102 as appropriate.
In general, a “processor” is logic circuitry that responds to and processes the basic instructions that drive a computer, and is also the term that is often used as shorthand for the central processing unit (CPU). The processor in a personal computer or embedded in small devices is often called a microprocessor.
With regard to processor topology, and as used herein, the term “processor” specifically refers to a physical processor. A “physical processor” is an integrated circuit (IC)—sometimes called a “chip” or “microchip”—comprising a semiconductor wafer (“silicate”) on which numerous tiny resistors, capacitors, and transistors form at least one processor core comprising at least one logical processor. Each processor core has the capability to execute system instructions, and each logical processor represents the hyperthreading capabilities (also known as symmetric multi-threading or “SMT”) by which a single processor core seemingly executes two threads in parallel (and thus appears to be two cores to the system).
Each physical processor is fixed into a single socket on a CPU motherboard. A physical processor may have more than one processor core (each having one or more logical processors). Each processor core will typically have its own level-1 cache but share a level-2 cache with other processor cores on the physical processor.
A “multi-core processor” is a physical processor having two or more cores for enhanced performance, reduced power consumption, and/or more efficient simultaneous processing of multiple tasks (e.g., parallel processing). For example, a “dual-core processor”—which, as its name suggests, is a multi-core processor having two processor cores—is somewhat similar to having two separate processors installed in the same computer. However, these two cores reside on a single physical processor and are essentially plugged into the same socket, and thus the connection between these two processor cores is faster than it would be for two single-core processors plugged into separate sockets.
Because of these performance gains, multi-core processing is growing in popularity as single-core processors rapidly reach the physical limits of possible complexity and speed. Companies that have produced or are working on multi-core products include AMD, ARM, Broadcom, Intel, and VIA. Both AMD and Intel have announced that they will market dual-core processors by 2005.
NUMA (non-uniform memory access) is a method of configuring a node of physical processors in a multiprocessing system so that they can share memory locally, improving performance and the ability of the system to be expanded. NUMA is typically used in a symmetric multiprocessing (SMP) system that is a “tightly-coupled, share everything” system in which multiple processors working under a single operating system access each other's memory over a common bus or “interconnect” path. Ordinarily, a limitation of SMP is that as microprocessors are added, the shared bus or data path gets overloaded and becomes a performance bottleneck; however, NUMA adds an intermediate level of memory (node memory) shared among that node's microprocessors so that all data accesses do not have to travel on the main bus.
Referring again to
A NUMA node typically consists of four physical processors interconnected on a local bus to a shared memory (the “L3 cache”) all on a single motherboard. This unit can be added to similar units to form a symmetric multiprocessing system in which a common SMP bus interconnects all of the nodes. Such a system typically contains from 16 to 256 microprocessors. To an application program running in an SMP system, all the individual processor memories look like a single memory.
When a processor core looks for data at a certain memory address, it first looks to its L1 cache, then on the L2 cache for the physical processor, and then to the L3 cache that the NUMA configuration provides before seeking the data in the “remote memory” located near the other microprocessors. Data is moved on the bus between the clusters of a NUMA SMP system using scalable coherent interface (SCI) technology. SCI coordinates what is called “cache coherence” or consistency across the nodes of the multiple clusters.
Exposing Processor Topology
Various embodiments of the present invention are directed to systems and methods for making a guest operating system aware of the topology of the subset of host resources currently assigned to it. For certain of these embodiments, at virtual machine boot time a Static Resource Affinity Table (SRAT) will be used by the virtualizer to group guest physical memory and guest virtual processors into virtual nodes. Thereafter the host physical memory behind a virtual node can be changed by the virtualizer as necessary, and the virtualizer will provide physical processors appropriate for the virtual processors in that node. This approach allows NUMA-aware operating systems executing on the virtual machine to schedule for optimal performance without further modification.
For certain alternative embodiments, the virtualizer may also provide dynamic processor topology information for the guest operating system in virtual machine memory. This information may be placed directly into the guest operating system's internal tables or, alternately, the guest operating system may execute additional code to pick this information from a shared memory location. The latter approach, referred to as “disclosing” (where the VM discloses information on a regular basis to the guest operation system, and the guest operating system regularly checks for updated info and adjusts accordingly) requires that the guest operating system be provided with additional code to cause it to periodically acquire this dynamic information.
For certain embodiments of the present invention, the guest operating system would execute a virtual machine call (a call to the virtualizer) which designates a virtual machine's physical memory page to be shared by both the virtualizer and the guest OS. This page may contain a control field with disclosure data to determine, for example: (a) whether the virtualizer should send an interrupt to the guest operating system whenever it changes the virtual machine topology to match changes in host computer system resource allocations to said virtual machine; (b) the vector to be used for the notification interrupt; (c) a generation counter which is incremented whenever the hypervisor updates the topology data; (d) a bit-mask of all virtual processors in the same SMT or hyperthreaded processor core; and/or (e) a bit-mask of all virtual processors in the same physical processor, that is, all logical processors in all cores in each physical processor. In addition, disclosure data may address any of the following aspects of efficiency: (a) thread priority; (b) I/O priority; (c) range of protected memory; (d) NUMA nodes; (e) data pertaining to near memory and far memory access; (f) processor speed and processor power consumption; (g) sockets and, for each core, hyperthreading; and/or (h) sharing level for each physical processor.
The scheduler of an operating system that has access to dynamic processor and NUMA topology information, such as when the “disclosing” approach is used, is able to use this information to optimize its own resource allocation mechanisms (e.g. processor scheduling, memory allocation, etc.) and resource utilization schemes. For certain additional embodiments of the present invention, the guest OS (either through virtualizer/virtual machine calls or through a shared memory page) may provide hints about resource allocation preferences to the virtualizer in a process called “hinting” (which is the logical converse of disclosing”). For example, if the guest OS would prefer to keep two virtual processors assigned to two cores on the same processor or two processors within the same NUMA node for efficiency, it could provide such a hint to the VM and the virtual machine scheduler could take this hint into account with regard to the virtualized processors as they pertain to the underlying physical processors assigned to said VM at any given time. More specifically, such hints may address any of the following aspects of efficiency: (a) thread priority; (b) I/O priority; and/or (c) latency information. Thus, for embodiments of the present invention, both the scheduler for the guest operation system as well as the scheduler for the VM—which independently manage resources—to utilize and employ “disclosing” and “hinting” to work cooperatively to maximize the efficiency of the entire system.
The various systems, methods, and techniques described herein may be implemented with hardware or software or, where appropriate, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention. In the case of program code execution on programmable computers, the computer will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. One or more programs are preferably implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language, and combined with hardware implementations.
The methods and apparatus of the present invention may also be embodied in the form of program code that is transmitted over some transmission medium, such as over electrical wiring or cabling, through fiber optics, or via any other form of transmission, wherein, when the program code is received and loaded into and executed by a machine, such as an EPROM, a gate array, a programmable logic device (PLD), a client computer, a video recorder or the like, the machine becomes an apparatus for practicing the invention. When implemented on a general-purpose processor, the program code combines with the processor to provide a unique apparatus that operates to perform the indexing functionality of the present invention.
While the present invention has been described in connection with the preferred embodiments of the various figures, it is to be understood that other similar embodiments may be used or modifications and additions may be made to the described embodiment for performing the same function of the present invention without deviating there from. For example, while exemplary embodiments of the invention are described in the context of digital devices emulating the functionality of personal computers, one skilled in the art will recognize that the present invention is not limited to such digital devices, as described in the present application may apply to any number of existing or emerging computing devices or environments, such as a gaming console, handheld computer, portable computer, etc. whether wired or wireless, and may be applied to any number of such computing devices connected via a communications network, and interacting across the network. Furthermore, it should be emphasized that a variety of computer platforms, including handheld device operating systems and other application specific hardware/software interface systems, are herein contemplated, especially as the number of wireless networked devices continues to proliferate. Therefore, the present invention should not be limited to any single embodiment, but rather construed in breadth and scope in accordance with the appended claims.
Finally, the disclosed embodiments described herein may be adapted for use in other processor architectures, computer-based systems, or system virtualizations, and such embodiments are expressly anticipated by the disclosures made herein and, thus, the present invention should not be limited to specific embodiments described herein but instead construed most broadly. Likewise, the use of synthetic instructions for purposes other than processor virtualization are also anticipated by the disclosures made herein, and any such utilization of synthetic instructions in contexts other than processor virtualization should be most broadly read into the disclosures made herein.