Cray: Faster Than A Bottleneck Bullet (BYTE Article)
Cray Research started its Business Systems Division about four years ago to build supercomputer systems that would serve business needs. The company's Cray CS6400 Enterprise Server uses an SMP architecture to support up to 64 SuperSparcs, each clocked at 60 or 85 MHz.
Since bus traffic normally degrades an SMP design that's built around eight processors, the CS6400 seems to defy logic. The system dodges this limit by using four buses instead of one to reduce memory contention. Each bus has its own private bank of RAM, while the processors connect to every bus, as shown in the figure .
Physical memory is interleaved among all four memory banks to create a shared memory space. You can add up to 64 memory modules to the system, for a maximum of 16 GB of RAM. To reduce bus traffic further, every processor module (which consists of four processors) has a 2-MB static RAM (SRAM) cache. Bus-watcher logic snoops all memory accesses and updates the caches to maintain memory coherency.
To ensure optimum I/O throughput, up to 16 buses support up to 64 I/O controllers. The I/O subsystem can manage up to 5 TB of on-line storage. Larger storage capacity is possible via the use of either disk arrays or an off-line archival medium, such as tape. The CS6400 supports communications via Ethernet, 100-Mbps Ethernet, FDDI/CDDI, Token Ring, and ATM.
To host mission-critical applications, a System Service Processor (SSP) -- which is actually a built-in Sun workstation -- monitors the system's health. In the event of a major failure, the SSP restarts the system and logically removes the problematic har dware. The system design allows you to hot-swap faulty hardware without interrupting service.
The computer runs Solaris 2.4, which is a Unix S5R4 derivative from Sun Microsystems. It can host Sun workstation applications and provide access to database applications, such as Oracle's Oracle7 and Informix Software's Informix Online Dynamic Server. The alliance with Sun also fits into a strategy of offering workstation solutions (via Sun), with connectivity to the CS6400 to tackle the huge jobs.
A basic Cray CS6400 has one processor module and 512 MB of RAM (but no disk) and costs $400,000. The basic system is rated at 3.8 GFLOPS, although floating-point calculations don't normally figure into the type of work this system does. The big box, with up to 64 processors and 16 GB of RAM, costs about $4 million.
The CS6400 is rather like the SPARCcenter-2000 in principle, but there are some significant differences:
- a quad-XDBus design:
Judging from the pictures (especially the large number of bus-watcher chips per board), each system board connects to all four XDBuses.
- XBus/XDBus runs at 55MHz (possibly 50MHz in early models).
- 'Custom format' XBus CPU modules - with multiple pin-grid connectors per module rather than the otherwise typical MBus connector.
- Maximum of 64 SuperSPARC CPUs: upto 16 system-boards each with upto 4 CPU modules.
- Mainframe-like "system-domains" capability, similar to the UltraSPARC-based Sun E10000 (StarFire).
- The chassis fans are probably under software control...!
The CS6400 cannot use standard MBus/XBus modules, but rather must use CS6400-specific modules (the module PCB layout is completely different). Confusingly, the CS6400-format CPU modules share the same names as the equivalent 'normal-format' modules: standard CPU options for the CS6400 list "SM51", "SM61", "SM51-2", "SM61-2" modules, with "SM81-2" listed as a standard upgrade.
The latter implies that even the hottest SuperSPARC CPU modules would not overheat any part of the CS6400.
You cannot use HyperSPARC, SM100 or CYM6001K CPU modules in the CS6400.
The CS6400 does "dynamic PROM selection" (within each domain!) in a similar manner to the SPARCcenter-2000, so the issues regarding having the same PROM versions on each system-board within a domain obviously apply.
Cray Superserver 6400CS6400
Data Center Cabinet
PART # BSD PART # DESCRIPTION 370-2732 420-6146-001 Microcom Modem 370-2733 429-6007-001 Blower Assembly, 60Hz 370-2734 429-6007-002 Blower Assembly, 60Hz 370-2735 429-6008-001 AC Line Filter / Main Breaker, 50Hz 370-2736 429-6009-001 AC Line Filter / Main Breaker, 60Hz 370-2737 429-6009-002 AC Line Filter / Main Breaker, 60Hz 370-2738 435-6000-007 I/O Cabinet PDU, 60Hz 370-2739 435-6000-008 I/O Cabinet PDU, 50Hz 370-2740 612-6002-001 Centerplane 370-2741 613-6003-003 Control Board 1 370-2742 613-6003-004 Control Board 2 370-2743 613-6004-004 System Board 2 370-2744 655-6103-001 Blower Cable, Front and Rear 370-2745 671-6128-001 Hot Swap Power Module 370-2746 421-6011-001 8MB SIMM Module 370-2747 421-6012-001 32MB SIMM Module 370-2748 420-6049-001 RS232 Serial Port PC Card 370-2749 420-6049-002 RS232 Serial Port PC Card 370-2750 420-6045-001 HIPPI Interface CHI Systems 370-2751 420-6081-001 Exabyte EXB-8505 Tape Drive 370-2752 420-6081-002 Exabyte EXB-8505 Tape Drive 370-2753 420-6144-001 QIC 1/4" Tape Drive 370-2754 421-6044-001 Dual Attach FDDI Interface 370-2755 655-6010-001 Tape Chassis SCSI Cable 370-2756 422-6118-003 68-Pin Differential SCSI Cable, 6M 370-2757 422-6150-001 External Remote Power Control Cable 370-2758 KIT85 EMI Material, SuperSPARC II Upgrade 370-2759 408-6007-002 Blower Airvane Switch Sensor 370-2760 612-6061-010 60MHz SuperSPARC Module 370-2761 612-6162-002 60MHz SuperSPARC Module 370-2762 612-6175-001 85MHz SuperSPARC II Module 370-2763 409-6011-001 Tape Chassis 200W Power Supply 370-2764 409-6012-001 Disk Tray 300W Power Supply 370-2765 420-6063-001 2.9GB Dual Port Disk Drive 370-2766 420-6113-001 2.9GB Dual Port Disk Drive 3 370-2767 420-6114-002 Dual Port Disk Tray 370-2768 420-6114-002 Enhanced Dual Port Disk Tray 370-2769 420-6116-001 Tape Drive Chassis 370-2770 420-6117-001 9GB Single Port Disk Drive 370-2771 420-6117-002 9GB Single Port Disk Drive 370-2772 420-6170-001 Single Port Disk Tray 370-2773 420-6170-002 Enhanced Single Port Disk Tray 370-2774 612-6010-001 ECB Assembly, Seagate Elite-9 Adapter 370-2775 655-6011-001 SCSI Disk Drive Port-A Cable 370-2776 655-6012-001 SCSI Disk Drive Port-B Cable 370-2777 655-6013-001 Single Port SCSI Disk Drive Cable 370-2778 420-6126-001 Differential SCSI Terminator 370-2779 422-6124-001 System Service Processor Interface Cable 370-2780 429-6005-001 Backplane Assembly, Power Supply 370-2781 CS3T2-020191 10-inch Fan 370-2782 435-6000-010 Circuit Breaker, Main, 250V 370-2783 435-6000-012 Circuit Breaker, Fan, 415V 370-2784 435-6000-013 Circuit Breaker, Fan, 415V, 62DL 370-2785 436-6000-003 Air Filter, 9.5" × 22.75" × 0.43" 370-2786 SP-16ASYN-01 16 Port Asynchronous Interface 370-2787 SP-ATM100-01 4615 SBus with 100MB ATM Adapter 370-2788 SP-ATM155-01 4615 SBus with 155MB ATM Adapter 370-2789 SP-DLTLM5-01 Tape Library 370-2790 SP-DLT450-01 Table-Top Tape Library 370-2791 SP-F3480S-01 1/2" Desktop Cartridge Tape Drive 370-2792 SP-CBL005-01 Differential SCSI Cable, 20' 370-2793 SP-CBL007-01 Differential SCSI Cable, 10' 370-2794 SP-XTBASE-01 HMX NCD Xterminal Base 370-2795 SP-XTMON-01 RHMX NCD Xterminal Monitor 370-2796 SP-XTKEY-01 HMX NCD Xterminal Keyboard 370-2797 SP-XTMSE-01 RHMX NCD Xterminal Mouse 370-2798 SP-EX903-01 EX480 TMCS Card 370-2799 SP-EX904-01 Exabyte EXB-480 Long Axis Belt 370-2800 SP-EX905-01 Exabyte EXB-480 Front Panel Assembly 370-2801 SP-EX906-01 EXB-480 Robotic Arm w Barcode Reader 370-2802 SP-EX907-01 Exabyte EXB-480 CHM Flex Cable 370-2803 SP-EX908-01 Exabyte EXB-480 Power Supply and Fan 370-2804 SP-EX909-01 Exabyte EXB-8505SL Drive with Carrier 370-2806 409-6010-005 Power Supply 1 370-2807 409-6010-006 Power Supply 2
- Not compatible with 85MHz SuperSPARC II.
- Compatible with 60MHz SuperSPARC and 85MHz SuperSPARC II.
- Seagate 992004-001 (firmware level 0105) works with CS6400, but Seagate 992004-003 (firmware level 2100) fails. See details on 370-2766 or Bug ID 4418573.
Last changes: Saturday, October 10, 2009 09:12:50 PM,
:P 2003 filibeto.org, site statistics