|Publication number||US7788334 B2|
|Application number||US 11/768,784|
|Publication date||Aug 31, 2010|
|Filing date||Jun 26, 2007|
|Priority date||Jun 26, 2007|
|Also published as||US20090006546|
|Publication number||11768784, 768784, US 7788334 B2, US 7788334B2, US-B2-7788334, US7788334 B2, US7788334B2|
|Inventors||Matthias A. Blumrich, Dong Chen, Alan G. Gara, Mark E. Giampapa, Philip Heidelberger, Martin Ohmacht, Valentina Salapura, Burkhard Steinmacher-Burow, Pavlos Vranas|
|Original Assignee||International Business Machines Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (4), Referenced by (7), Classifications (7), Legal Events (5)|
|External Links: USPTO, USPTO Assignment, Espacenet|
The U.S. Government has a paid-up license in this invention and the right in limited circumstances to require the patent owner to license others on reasonable terms as provided for by the terms of Contract. No. B554331 awarded by the Department of Energy.
The present invention is related to the following commonly-owned, co-pending United States Patent Applications filed on even date herewith, the entire contents and disclosure of each of which is expressly incorporated by reference herein as if fully set forth herein. U.S. patent application Ser. No. 11/768,777, for “A SHARED PERFORMANCE MONITOR IN A MULTIPROCESSOR SYSTEM”; U.S. patent application Ser. No. 11/768,645, for “OPTIMIZED COLLECTIVES USING A DMA ON A PARALLEL COMPUTER”; U.S. Pat. No. 7,694,035, for “DMA SHARED BYTE COUNTERS IN A PARALLEL COMPUTER”; U.S. patent application Ser. No. 11/768,697, for “A METHOD AND APPARATUS OF PREFETCHING STREAMS OF VARYING PREFETCH DEPTH”; U.S. patent application Ser. No. 11/768,532, for “PROGRAMMABLE PARTITIONING FOR HIGH-PERFORMANCE COHERENCE DOMAINS IN A MULTIPROCESSOR SYSTEM”; U.S. patent application Ser. No. 11/768,857, for “METHOD AND APPARATUS FOR SINGLE-STEPPING COHERENCE EVENTS IN A MULTIPROCESSOR SYSTEM UNDER SOFTWARE CONTROL”; U.S. Pat. No. 7,669,012, for “INSERTION OF COHERENCE EVENTS INTO A MULTIPROCESSOR COHERENCE PROTOCOL”; U.S. patent application Ser. No. 11/768,791, for “METHOD AND APPARATUS TO DEBUG AN INTEGRATED CIRCUIT CHIP VIA SYNCHRONOUS CLOCK STOP AND SCAN”; U.S. patent application Ser. No. 11/768,795, for “DMA ENGINE FOR REPEATING COMMUNICATION PATTERNS”; U.S. Pat. No. 7,680,971, for “METHOD AND APPARATUS FOR A CHOOSE-TWO MULTI-QUEUE ARBITER”; U.S. patent application Ser. No. 11/768,800, for “METHOD AND APPARATUS FOR EFFICIENTLY TRACKING QUEUE ENTRIES RELATIVE TO A TIMESTAMP”; U.S. Pat. No. 7,701,846, for “BAD DATA PACKET CAPTURE DEVICE”; U.S. patent application Ser. No. 11/768,593, for “EXTENDED WRITE COMBINING USING A WRITE CONTINUATION HINT FLAG”; U.S. patent application Ser. No. 11/768,805, for “A SYSTEM AND METHOD FOR PROGRAMMABLE BANK SELECTION FOR BANKED MEMORY SUBSYSTEMS”; U.S. patent application Ser. No. 11/768,905, for “AN ULTRASCALABLE PETAFLOP PARALLEL SUPERCOMPUTER”; U.S. patent application Ser. No. 11/768,810, for “SDRAM DDR DATA EYE MONITOR METHOD AND APPARATUS”; U.S. patent application Ser. No. 11/768,812, for “A CONFIGURABLE MEMORY SYSTEM AND METHOD FOR PROVIDING ATOMIC COUNTING OPERATIONS IN A MEMORY DEVICE”; U.S. patent application Ser. No. 11/768,559, for “ERROR CORRECTING CODE WITH CHIP KILL CAPABILITY AND POWER SAVING ENHANCEMENT”; U.S. patent application Ser. No. 11/768,552, for “STATIC POWER REDUCTION FOR MIDPOINT-TERMINATED BUSSES”; U.S. patent application Ser. No. 11/768,527, for “COMBINED GROUP ECC PROTECTION AND SUBGROUP PARITY PROTECTION”; U.S. patent application Ser. No. 11/768,669, for “A MECHANISM TO SUPPORT GENERIC COLLECTIVE COMMUNICATION ACROSS A VARIETY OF PROGRAMMING MODELS”; U.S. patent application Ser. No. 11/768,669, for “MESSAGE PASSING WITH A LIMITED NUMBER OF DMA BYTE COUNTERS”; U.S. Pat. No. 7,738,443, for “ASYNCRONOUS BROADCAST FOR ORDERED DELIVERY BETWEEN COMPUTE NODES IN A PARALLEL COMPUTING SYSTEM WHERE PACKET HEADER SPACE IS LIMITED”; U.S. patent application Ser. No. 11/768,682, for “HARDWARE PACKET PACING USING A DMA IN A PARALLEL COMPUTER”; and U.S. patent application Ser. No. 11/768,752, for “POWER THROTTLING OF COLLECTIONS OF COMPUTING ELEMENTS”.
The invention relates to parallel computer system, and more particularly relates to a remote messaging engine capable of supporting/sending multiple remote messages to multiple remote nodes comprising a parallel computer network of interconnected compute nodes, without need for compute node processor control of the remote messaging, triggered by a single send message from a source node. The remote messaging engine for multiple node remote messages, and the novel messaging operation provided thereby is set forth and described herein for the purpose of conveying the broad inventive concepts. The drawings and descriptions provided are not meant to limit the scope and spirit of the invention in any way.
Parallel computer systems include multiple compute nodes that each run threads of a global application program to accomplish tasks or processes, e.g., BlueGene/P ultrascalable Petaftop Parallel Supercomputer, by IBM Corporation. The individual compute nodes, and the instances of the global application running at the compute node carry out message passing to complete designated node-specific portions of the task or process. During message passing, an event is generated each time a message is received at a compute node comprising a parallel computer system. The compute node (and local processors) processes such events according to its inherent event-processing algorithm. In general, various devices often use a special type of event processing system for managing various messages. That is, to ensure such communication between various compute nodes comprising a parallel computer system, a standard known as message passing interface (MPI), defined by a group of organizations including various vendors and researchers is used.
MPI is the industry-standard message-passing interface. An MPI program consists of autonomous processes, executing their own code, which need not be identical. Typically, each process or application communicates via calls to MPI communication primitives, where each process executes in its own and shared memory. Such message passing allows the local processors comprising the compute node, and applications running thereon (a thread or instance of the global application or process) to cooperate with each other. Generally speaking, an MPI is an interface designed to allow a user to code data such that the local processors at the compute nodes comprising the network are able to send and receive data and information to coordinate the disposition of the global application or process. MPI is available on a wide variety of platforms ranging from massively parallel systems (IBM, Cray, Intel Paragon, etc.) to networks of workstations.
The use of Direct Memory Address (DMA) technology provides for reducing CPU (processor) workload in the management of memory operations required for messaging in any computer system, and are particularly relied on in large parallel computer systems. DMA, or DMA engines, work in conjunction with the local application or thread implementing the MPI application. Workload that would normally have to be processed by a CPU at a compute node is instead handled by the DMA engine. The use of DMA technology in large parallel computer systems is limited somewhat by such system's inherent need for tight hardware control and coordination of memory and message-passing operations. That is, required tight coupling between memory operations and CPU operations poses some challenges, including the need for a sending side compute node (a source compute node originating a message to be passed to another or multiple other compute nodes) to have awareness of the receiver node's remote address spaces, multiple protection domains, locked down memory requirements (also called pinning), notification, striping, recovery models, etc.
In parallel computer, such as the BlueGene/P, a “Rendezvous” protocol is often used to send long messages between compute nodes. Following the Rendezvous protocol, a source compute node (a thread or instance of the global application running on the sending side) sends a long message by first passing a request to send (RTS) packet to the target compute node. The RTS contains information or data identifying the source compute node and the message being sent, e.g., number of total bytes. The target compute node replies to the RTS by generating and sending a “CTS (clear to send)” packet, assuming the target compute node is able to receive. The CTS includes information or data describing the target compute node (destination side) in order to receive the entire message. Finally, the source compute node sends self-describing “data” packets to the Target node, which can arrive in any order. The packet transfer continues until the entire message has been sent. RTS packet transfers, or message transfers following the Rendezvous protocol, are ordered with respect to the transfer of other messages out of a compute node, or into a compute node, e.g., with respect to other rendezvous or eager messages.
As mentioned above, computer systems and in particular parallel computer systems such as BlueGene/P utilize DMA engines to asynchronously move data (e.g., message passing) between in-node memory and the communications network (other compute nodes). DMA engines operate under a set of constructs used by message passing libraries (as in MPI) to set up and monitor completion of DMA data transfers. In large parallel computer systems such as BlueGene/P, DMAs may be fabricated within or integrated into the same ASIC comprising the node's processors. As such, size is a consideration and therefore such DMAs often have finite resources, for example, byte counters to tracks the number of bytes sent or received in a DMA transfer, which must be managed wisely to maximize the exchange of the many messages (at the compute node comprising the DMA channel). In peak performance applications, many outstanding messages might be regularly outstanding that must be managed by the DMA engine before data transfers.
Message passing libraries used for DMA message transfer in parallel computer systems inefficiently implement known rendezvous protocol for some applications. Before the instant invention, it has been unknown for conventional parallel computer systems to operate with DMA engines that have inherently limited numbers of byte counters and other registers because of size constraints. Until the recent development of IBM's BlueGene/P ultrascalable Petaflop Parallel Supercomputer, which includes compute nodes with DMA engines integrated within a single ASIC core (and therefore a limited number of byte counters), efficiency utilizing such a limited number of byte counters was not a priority for computer designers. Hence, other versions of the rendezvous protocol to provide for a DMA engine's efficient management of a limited number of counters are unknown, and would be desirable for use in a supercomputer such as BlueGene/P.
For that matter, commonly-owned co-pending U.S. patent application Ser. No. (YOR820070343), entitled: DMA Shared Byte Counter In A Parallel Computer, filed concurrently and incorporated by reference herein, discloses a DMA engine for use in a parallel computer system, a method for passing messages using such a DMA engine in a parallel computer system and a parallel computer system utilizing the novel DMA engine for sharing of byte counters by multiple messages. The aforementioned DMA Shared Byte Counter, however, is unable to determine (the application thread running at a compute node and using the local DMA engine for message-passing) whether a message that is using a shared byte counter has been completed other than at the time when it can be determined that all of the messages (message packet transfers) sharing the shared byte counter have completed.
Direct memory access (DMA) allows certain hardware sub-systems within a computer or computer system to access system memory for reading and/or writing independent of the central processing unit, or multiple central processing units in the case of parallel computers and computer systems. DMA use is made by disk drive controllers, graphics cards, network cards, sound cards and like devices. Computer systems that employ DMA channels can transfer data to and from devices with much less CPU overhead as compared to computer systems without a DMA channel.
A DMA transfer comprises copying a block of memory from one device to another (comprising the computer system). The CPU initiates the DMA transfer, but the DMA carries out the task. For what is known in the art as “third party” DMA, for example, as used in conjunction with conventional ISA bus operation, a DMA controller or engine that is normally part of the motherboard chipset performs the transfer. For example, the BlueGene/P, a parallel multi-computer system by International Business Machines (IBM), includes a DMA engine integrated onto the same chip as the processors (CPUs), cache memory, memory controller and network logic.
DMAs are used conventionally to copy blocks of memory from system RAM to or from a buffer on the DMA device w/o interrupting the processor, which is quite important to high-performance embedded systems. DMA is also used conventionally to offload expensive memory operations, such as large copies from the CPU to a dedicated DMA engine. For example, a “scatter gather” DMA allows the transfer of data to and from multiple memory areas in a single DMA transaction. Scatter gather DMA chains together multiple simple DMA requests in order to off-load multiple input/output interrupt and data copy tasks from the processor or CPU.
In a DMA engine for transferring data from a network interface the CPU provides destination address for moving data from the network interface to the memory. Length of packets or their semantics received from the network interface are not know in advance. Multiple packets can contain various parts of a single data message/transfer, such as in MPI messages. A DMA engine moves received packets to destination addresses without Packet reordering if received out-of-order. Storing packets in a single continuous address space if packets are part of a single message. Some network protocols such as MPI with rendezvous protocol support acknowledgement means to allow to the other party transfer of large messages using a number of packets. Destination address for the whole data block in a continuous address space at the receiver side provided by the receiver ahead of time/at the beginning of transmission to the transmitter side.
What would be desirable in the field of parallel computer systems and their design, and in particular in parallel computer systems including DMA engines constructed to perform remote message sends to remote compute nodes comprising the parallel computer system automatically in hardware without core processor (e.g., CPU) involvement, triggered by a single message from a source compute node.
This invention describes a parallel computer system comprising a network of interconnected compute nodes, where each compute node comprises a plurality of processors, a memory and direct memory access engine (DMA) constructed for passing at least one remote message triggered by a first single source compute node (A) sending a single message. The DMA engines of the interconnected compute nodes comprise a processor interface for communicating with said plurality of processors, DMA logic; injection and reception byte counters; injection and reception FIFO metadata; a memory interface and a DMA network interface. The parallel computer system further includes that the DMA engines provide for putting remote descriptors from the first single source compute node (A) into an injection FIFO at remote second compute node (B). The parallel computer system may include as well that the first single source compute node (A) implements a 3rd party send by sending a single remote first message including at least one remote message descriptor to a remote second compute node (B), to control the remote second compute node (B) to send a second data message, and wherein remote second compute node (B) sends said second data message to third compute node (C).
Moreover, the parallel computer system may include that the first compute node (A) implements at least one 3rd party send by sending a single remote get message including at least one remote message descriptor to a remote second compute node (B), which upon receipt controls the DMA engine to pass at least one data message corresponding to the at least one remote message descriptor. The parallel computer system may include that the wherein remote second compute node (B) sends said multiple data messages to a third remote compute node (C), and/or that the remote second compute node (B) sends said multiple data messages to the first compute node (A), and/or that the remote second compute node (B) sends multiple data messages to multiple compute nodes, where each of the multiple data messages corresponds to each of multiple message descriptors.
The invention also includes a method for passing remote messages in a parallel computer system comprising a network of interconnected compute nodes, wherein a first compute node (A) sends a single remote message to a remote second compute node (B) in order to control said remote second compute node (B) to send at least one remote message. The method includes controlling a DMA engine at first compute node (A) to prepare the single remote message to include a first message descriptor and at least one remote message descriptor for controlling remote second compute node (B) to send at least one message, including putting the first message descriptor into an injection FIFO at the first compute node (A) and sending the single remote message and the at least one remote message descriptor to the second compute node (B).
The objects, features and advantages of the present invention will become apparent to one skilled in the art, in view of the following detailed description taken in combination with the attached drawings, in which:
The present invention provides for remote message passing operation in a distributed memory parallel computer system such as IBM, Inc.'s ultrascalable petaflop parallel supercomputer, as described in commonly-owned, pending U.S. patent application Ser. No. 11/768,905, filed concurrently herewith and entitled:
The ASIC nodes are interconnected by multiple independent networks that optimally maximize communications throughput and minimize latency. The multiple networks may include three high-speed networks for parallel algorithm message passing including a Torus with DMA (direct memory access), collective network, and a Global Asynchronous network that provides global barrier and notification functions. These multiple independent networks may be collaboratively or independently utilized according to the needs or phases of an algorithm for optimizing algorithm-processing performance.
At least four modes of operation are supported: virtual mode, SMP 1-core mode, SMP 4-core mode and dual mode. In the virtual node mode, each of the processing cores will perform its own MPI (message passing interface) task independently. Each core uses a quarter of the memory (L3 and DRAM) of the node, while coherence among the four MPI within the node and across the nodes is maintained by MPI. In the SMP (Symmetric Multi Processor) 1-core mode, one core performs a single MPI task, using the entire memory capacity of the node. In the SMP 4-core mode, one MPI task with 4 threads is running, using the whole node memory capacity. The fourth mode is called the dual mode. In this hybrid case, two SMP MPI tasks are running, with each SMP using 2 cores running a thread each. Finally, one can also support modes such as a 1, 3 split, and 1, or 2 or 3 cores idling. Thus a node can trade off amount of memory versus parallelism, a feature unique to this supercomputer.
Because of the torus's DMA feature, internode communications can overlap with computations running concurrently on the nodes. Also complex forms of messaging protocols, particular arithmetic functions, often called “reduction functions”, are required to be invoked on message data as it arrives. One computer node core may be implemented to perform these functions without distracting computations on other cores. Additionally, the application is able to define arbitrarily complex reduction functions, supporting new algorithmic development that overlaps computational power with communication activities. For particular classes of parallel algorithms, or parts of parallel calculations, this architecture may apply the power of some or all cores to work in collaboration on communication activities.
The novel ultrascalable petaflop parallel supercomputer architecture allows the multiple networks to be utilized simultaneously, independently, or in collaboration. Because of the system-on-a-chip level integration, in collaboration with operating system software designed to take advantage of this architecture, alternating between such modes of operation can occur with minimal overheads. In cases where the computation is awaiting message arrival, this may happen automatically.
To connect nodes, multiple network paradigms are implemented to interconnect nodes for use individually or simultaneously, and include three high-speed networks for parallel algorithm message passing. Additional networks are provided for external connectivity and are used for Input/Output, System Management and Configuration, and Debug and Monitoring services for the supercomputer nodes. The high-speed networks preferably include n-dimensional Torus, collective network, and Global Signal configurations. The use of each of these networks may switch back and forth based on algorithmic needs or phases of algorithms. For example, parts of calculations may be performed with the Torus, or part on the collective network, which facilitates the development of new parallel algorithms that simultaneously employ multiple networks in novel ways.
Each node comprises a single ASIC and a plurality of SDRAM-DDR2 memory chips. The nodes are interconnected through at least six (6) networks, the highest aggregate bandwidth of which is a nearest neighbor link that maps the nodes into a physical 3-dimensional torus. The interconnect structure comprises a torus with preferably no asymmetries as the compute nodes communicate with the same bandwidth and nearly the same latency to nodes that are physically close as to those which are physically located on neighboring racks. Such interconnect structure facilitates simpler programming models. The ASICs that power the nodes are based on system-on-a-chip (s-o-c) technology and incorporate all of the functionality needed by the system. The ASICs or compute nodes include 8 MB or more of extremely high bandwidth embedded DRAM. The compute nodes are physically small, allowing for a very high density of processing and optimizing cost/performance.
In a preferred embodiment, the system packaging comprises 512 processing nodes on a doubled-sided board or “midplane”. Each processing or compute node contains 4 processor cores for handling computation and message passing operations. In addition, associated with a prescribed plurality of processing nodes is a dedicated node that comprises a quad-processor with external memory, for handling of I/O communications to and from the compute nodes. Each I/O node has an operating system (Linux based) that can handle basic tasks and all the functions' necessary for high performance real time code. For compiling, diagnostics, and analysis, a host machine running a global application is required. The I/O compute nodes contain a software layer above the layer on the compute nodes for handling host communications. The choice of host compute node depends on the class of applications, their bandwidth needs and performance requirements.
As used herein, two packet types are distinguished, general packets and special packets. General packets are handled by DMA engines at compute nodes comprising the parallel computer system similar to conventional transfers, and special packets are handled by the novel processing ability of the invention, and contain the destination address(es) where the special packets are to be stored. The invention detects special packets, extracts destination addresses from the packets and reorders the packets if received out-of-order. The novel parallel computer system, DMA engine, and method for message passing tracks the number of received packets, or the number of outstanding packets by using DMA channel byte counters.
Special packets are part of a long message (data transfer) partitioned into multiple packets which are to be stored into continuous address space for further handling by the application. Special packets contain a destination address on the receiver side as a part of the packet header, extended header or payload. The destination address is disclosed by the receiver during the rendezvous prior to the message start or at the beginning of data transfer. Special packets are identified by one or more bits in the packet header. For that matter, the packet header contains one or more fields to specify packet type (Dm): 0—general packet; and 1—special packet. Special packets contain several bytes to specify the destination address, or address offset in the receiver address space, “Put offset” fields, packet header, extended header or payload contain information on channel, and VC fields, and are constructed to support one or more channels to keep track of one or more data streams. In preferred embodiments the parallel computer system, DMA engine, and novel message passing operation of the invention include that one or more DMA channels for data reception are supported, and can receive several intermixed packet streams by keeping track of number of received and outstanding packets belonging to a single message/packet stream. This is implemented using injection and reception byte counters in each DMA channel, which are set at the beginning of the reception of a packet stream at the remote compute node. The reception or injection byte counters use a specified number of bytes, or specified number of packets that will be received. That is, at the reception of a packet, a channel field in the packet identifies the channel and the corresponding channel counter is updated (decremented by the number of valid data bytes in the package). When the channel counter reaches 0 all packets/bytes have been received. The channel then notifies the processor by raising the interrupt or uses some other means for processor notification (like polling).
In more detail, compute node or ASIC 102(1) may function as both a compute node and an I/O node in the parallel computer system (100). Compute node 102(1) comprises a plurality of processors or processor cores, 110(1), . . . 110(p), but preferably four (p=4). Each of the processor cores (110(p)) includes a “double” floating point unit that includes two coupled standard floating point units. This arrangement gives a peak performance of four floating point operations per processor core per clock cycle. Preferably, each processor core comprises a PowerPC450 embedded core available from IBM microelectronics. The skilled artisan should note that future versions of this core may be used as technology improves, without deviating from the scope and spirit of the invention. A description of the functionality of such a PowerPC450 core may be found at http://www.ibm.com/chins/power/powetpc/.
Besides the embedded processing cores 110(p), and floating point cores (not shown in
DMA 116 directly controls transfer of long messages, which long messages are typically preceded by short protocol messages that are deposited into reception FIFOs on a receiving node (for example, compute node 102(2)). Through these protocol messages, the sender compute node and receiver compute node agree on which injection counter (130) and reception counter (134) identifications to use, and what the base offsets are for the messages being processed. Long message transfer may be initiated by a core processor on the sending node by placing a “put” message descriptor into an injection FIFO 118 (in memory 114), writing the injection counter base and value via writes via the DMA (processor) interface 122, and appropriately modifying the injection FIFO descriptor 132 for that message. This includes advancing a tail pointer indicating the “last” message descriptor in the via a “write” to the DMA processor interface 122. DMA logic 124 reads the injection FIFO descriptor 132 and recognizes which FIFOs have messages to be sent.
The DUMA logic causes the memory interface 126 to read the descriptor in the Injection FIFO 118. The put message descriptor includes the injection (130) and reception counter (134) identifications to be used, the message length, the initial injection and reception offsets of the message, the destination node and other network routing information. The DMA 116 begins fetching the message and assembling it into packets to be “put” on to the network (108). Each packet contains an offset from the reception counter (134) where the data from this packet is to be stored, and a count of how many bytes in this packet should be written. DMA 116 is responsible for updating this information correctly for each packet, and puts the packets into the network interface 128 (when space is available), at which time the packet enters the network and is routed to the destination compute node (e.g., compute node(p)).
After DMA 116 puts the message in the network interface 128, it decrements the specified injection counter 130 by the number of bytes in the packet. Upon reaching the destination, the packet is put into the network interface at that compute node (e.g., 102(p), and the node's local DMA “recognizes” that the packet is there. The receive compute node DMA reads the reception counter id, offset and count from the received packet, looks up the reception counter base address, writes the appropriate number of bytes starting at the base plus packet offset, and then decrements the counter value by the bytes.
If a remote get operation is used, instead of the processor on the sending node injecting a descriptor into the injection 118, the receiving node sends a short get message (which contains a put descriptor) to the sender compute node (e.g., 102(p)), and the DMA logic at the sender compute node puts this descriptor into its associated injection FIFO, and advances that FIFO's pointers appropriately. To share a byte counter, the base address of the shared counter must be set to a value smaller than the base address of any message to be using that counter. The initial value of the counter is set to zero. The initial offset in a message descriptor is the message's starting address minus this base offset. The particular processor increments the counter value by the current message length, and in accordance with the novel operation, said processor need only know the current message length, but not the lengths of the other messages using this counter, nor the number of bytes that have already been received.
Network 108 preferably displays a 3-dimensional torus functionality, providing all the network link cut-through routing buffers and routing control block that allow any two nodes to communicate with low latency. The four (or “p”) processor cores embedded in ASIC (node 102(1)) as shown may be utilized for message handling and computation operations. Virtual cut-through torus routing is supported in a hardware block, which is integrated into the compute nodes (102(p)) to allow for the elimination of the network adapter, typically required in conventional parallel computer system operation. Preferably, a virtual channel routing network is supported with two (2) dynamic and two (2) deterministic channels.
The same compute node ASIC construction is used as an I/O node, which is associated with a subset of the compute nodes (e.g., 16, 32, 64, or 128 compute nodes), for handling fileserver communication and I/O operations. The only difference between an I/O compute node and a computation compute node is that an I/O node enables and uses an external network interface, such as the 10 Gigabit Ethernet (not shown). While the compute nodes may have the integrated 10 Gb Ethernet (they share a common ASIC), for purposes of discussion, the 10 Gb Ethernet interface is enabled at the I/O nodes only. The 10 Gb Ethernet functional units (XEMAC, ToMAL) interfaces to an arbiter, which arbitrates inputs to L3 from torus DMA 116 and from the 10 Gb Ethernet (not shown). This allows the processor within a node to set up a transfer and return to handling messages with very little overhead.
As indicated herein, it should be understood that the present invention could be realized in hardware, software, or a combination of hardware and software, such as in the ultrascalable petaflop parallel supercomputer described above. The invention, however, is not limited to applications therein, but may be implemented in any kind of parallel computer/server system(s)—or other parallel computer system adapted for carrying out the novel message passing network interface for repeat communications described herein. A typical combination of hardware and software could be any general-purpose multiple parallel computer system or network with a computer program that, when loaded and executed, carries out the respective methods described herein. Alternatively, a specific use parallel computer system, or linked computer network, containing specialized hardware for carrying out one or more of the functional tasks of the invention, could be utilized.
For that matter, the present invention can also be embodied in a computer program product, which comprises all the respective features enabling the implementation of the methods for utilizing the network interface described herein, for example, the exemplary methods depicted in figures herein, and which product—when loaded in the parallel computer system or network, is able to carry out these and related methods. Computer program, software program, program, or software, in the present context mean any expression, in any language, code or notation, of a set of instructions intended to cause a parallel computer network or system having an information processing capability to perform a particular function either directly or after either or both of the following: (a) conversion to another language, code or notation; and/or (b) reproduction in a different material form.
Each FIFO (injection or reception) comprises 4 FIFO registers, as shown in
Upon initialization, the head and tail pointer registers 133(3), 133(4) of an injection FIFO 132 are set to the start address contained in register 133(1). The thread or instance of the global application running at the compute node write injection descriptors into the injection FIFO 118 and move the tail pointer in tail pointer register 133(4). The DMA 116 detects that the head and tail points (as maintained in head (133(3)) and tail (133(4))) registers are now different. The DMA therefore begins processing the descriptors maintained in the FIFO. Each injection descriptor is 32 bytes long, and corresponds to one message to be sent over the network (injection descriptors are essentially messages, which identify data for transfer by the message). Once a message is sent, the DMA moves the head pointer up by 32 bytes (in head pointer register 122(3), until it reaches the value maintained in the tail pointer register 133(4). When either the head or tail pointer reaches the end address maintained by register 133(2), they wrap back to start address (in start address register 133(1)). The thread or instance of the global application program “wraps” back the tail pointer, and the DMA 116 automatically, and without processor interaction (or interruption) controls the head pointer wrap back.
The valid descriptors are stored in memory 114 within memory addressed extending (1) from the head pointer to the tail pointer, if the number maintained in the tail pointer register (133(4)) is larger than the number maintained in the head pointer register (133(3)), or (2) if from the address represented by the number maintained in the head pointer register to the end address maintained in the end address register, then from the start address in the start address register to the tail pointer as maintained in the tail pointer register, if the number in the tail pointer register is less than the number maintained in the head pointer register.
The 16 byte network packet header 139(2) includes a torus network hardware header 139(21) containing 8 bytes, and an application header for memory FIFO packets or a DMA put/get hardware header 139(22). When sending over the torus network, a long message is broken into smaller packets. On Blue Gene/L and P, the torus network packet size ranges from 32 bytes to 256 bytes, in increments of 32 bytes. Torus network hardware header format (141) depicts in detail the breakdown of headers included in torus network hardware header 139(21). These include “1 byte marker” 141(1) that indicates the beginning of a data packet, where “10 bits routing info” 141(2) contains the torus routing information. A “3 bits-packet size” field 141(3) determines the packet size, and a “2 bits target node processor” field 141(4) maintains a processor core for a message. The destination node is determined by a “3 bytes target node (x,y,z) coordinates (141(6)). Because there are 4 processors on a Blue Gene/P compute chip, a 2 bit processor ID field is sufficient in the instant compute node embodiment. A 1 bit DMA flag 141(5) and the 8 byte DMA header 139 (22) are used by the receiving node to decode the packet.
The receiving DMA handles 3 types of packets, a memory FIFO packet, a put packet or a get packet. When the DMA flag 141(5) in the torus header is set to 0, the packet is a memory FIFO packet. The memory FIFO packet is put into one of the reception FIFOs on the receiving node. The reception FIFO mapping is done locally at the receiving node, depending on the status of “2 bits target node processor ID (flag) 141(4). A reception FIFO is handled in a manner similar to that of an injection FIFO, except that the data packets have sizes from 32 bytes to 256 bytes, and the DMA moves the tail pointer 133(4) and the application thread running on the processors (comprising the compute node) moves the head pointer 133(3).
The application header for memory FIFO packets or DMA put/get hardware header 139(22) comprise a FIFO packet application header format 143, or DMA put/get hardware header 144. The FIFO packet application header format 143 includes a “4 byte put offset (143(1)) and a 4 byte application header 143(2). Since a long message is broken into smaller packets, the “sending” DMA updates both the send message address offset 139(13) and 4 byte put offset 143(1) after each packet is sent. The packets can arrive at a destination node out of order. The 4 byte put offset 143(1) provides for reordering the packets for the receiving compute node processor(s) to reassemble a FIFO message. The 4 byte application header 143(2) is not used by the DMA for FIFO packets, and usually contains a function ID that the receiving node will execute once the packet is received.
When the 1 bit DMA flag 141(5) is set to 1 (in torus network hardware header 141), the FIFO packet application header format 143 is then replaced with the DMA put/get header format 144. Therein, if a remote get flag (144(4)) is set to 0, then the packet is a “put” packet, and 4 byte put offset 144(1) field, and 1 byte rdma counter ID (144(2)) field are used. The 4 byte put offset field (144(1) is the put offset in bytes, as is found in the FIFO packet. For each packet received, the receiving DMA will look up a reception counter, identified in the DMA put/get header format 144 by the 1 byte rdma counter ID (144(2)). The counter's base address 135(1) is added to the 4 byte put offset 144(1) to form the actual physical address of the packet's data payload or the receiving node. A “1 byte, number of valid bytes in payload” 144(3) contains the number of valid bytes in the payload. In typical long message transfers, the sending node sets up an injection counter and a receiving node sets up a reception counter with the total number of bytes in byte counter 131(2), and byte counter 135(2) to be transferred in the message via a remote put. The DMA 116, after sending/receiving each packet, will update the byte counter by subtracting the number of valid bytes in the packet data payload. When the byte count reaches 0, the message transfer is complete.
The 3rd DMA operating mode is remote get. In this case, a remote get flag 144(4) is set to 1, and 4 byte put offset 144(1) field, 1 byte rdma counter ID (144(2)) field are not used. On a sending node A, the injection DMA does the similar work as a remote put packet except that a message is fully contained in one torus packet. The data payload that node A sends to node B, contains a 32 byte descriptor for node B, to be put into a “1 byte, injection FIFO with FIFO ID (144(5)). Once the packet is received on node B, node B puts the descriptor contained in the packet payload into its injection FIFO, then sends the data related to this descriptor back to node A.
While the above-described is the standard mode RDMA remote get supported by the Blue Gene/P hardware, the novel construction also provides for new multiple node messaging operations, based in the ability to directly send a descriptor into a receiving node's injection FIFO. For that matter,
The first method of 3rd party send can be extended to a multiple party (that is, multiple node send as Node A could ask Node B to send a descriptor to Node C, resulting in Node C send data to Node D, etc.
The first and second method can also be combined to have node A send a single packet to node B, resulting in node B sending multiple messages to multiple targets.
The invention has been described herein with reference to particular exemplary embodiments. Certain alterations and modifications may be apparent to those skilled in the art, without departing from the scope of the invention. The exemplary embodiments are meant to be illustrative, not limiting of the scope of the invention.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4777595 *||Mar 24, 1986||Oct 11, 1988||Digital Equipment Corporation||Apparatus for transferring blocks of information from one node to a second node in a computer network|
|US6072781 *||Oct 22, 1996||Jun 6, 2000||International Business Machines Corporation||Multi-tasking adapter for parallel network applications|
|US20050076163 *||Nov 22, 2004||Apr 7, 2005||Broadcom Corporation||Flexible DMA descriptor support|
|US20060206635 *||Mar 10, 2006||Sep 14, 2006||Pmc-Sierra, Inc.||DMA engine for protocol processing|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US8321579 *||Jul 26, 2007||Nov 27, 2012||International Business Machines Corporation||System and method for analyzing streams and counting stream items on multi-core processors|
|US8983779||Feb 1, 2012||Mar 17, 2015||International Business Machines Corporation||RTM seismic imaging using incremental resolution methods|
|US9002651||Feb 1, 2012||Apr 7, 2015||International Business Machines Corporation||RTM seismic imaging without scratch data storage system and method|
|US9063248||Feb 1, 2012||Jun 23, 2015||International Business Machines Corporation||RTM seismic imaging using combined shot data|
|US20090031175 *||Jul 26, 2007||Jan 29, 2009||Charu Chandra Aggarwal||System and method for analyzing streams and counting stream items on multi-core processors|
|US20100064070 *||Mar 11, 2010||Chihiro Yoshimura||Data transfer unit for computer|
|US20150234744 *||Feb 18, 2014||Aug 20, 2015||National University Of Singapore||Fusible and reconfigurable cache architecture|
|U.S. Classification||709/212, 710/22, 709/213|
|International Classification||G06F13/28, G06F15/167|
|Jan 24, 2008||AS||Assignment|
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BLUMRICH, MATTHIAS A.;CHEN, DONG;GARA, ALAN G.;AND OTHERS;REEL/FRAME:020407/0245;SIGNING DATES FROM 20070809 TO 20070907
Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BLUMRICH, MATTHIAS A.;CHEN, DONG;GARA, ALAN G.;AND OTHERS;SIGNING DATES FROM 20070809 TO 20070907;REEL/FRAME:020407/0245
|Apr 15, 2008||AS||Assignment|
Owner name: U.S. DEPARTMENT OF ENERGY, DISTRICT OF COLUMBIA
Free format text: CONFIRMATORY LICENSE;ASSIGNOR:INTERNATIONAL BUSINESS MACHINES CORPORATION;REEL/FRAME:020809/0230
Effective date: 20080222
|Apr 11, 2014||REMI||Maintenance fee reminder mailed|
|Jul 11, 2014||FPAY||Fee payment|
Year of fee payment: 4
|Jul 11, 2014||SULP||Surcharge for late payment|