Survey of Interconnect Topologies
 Share of different Interconnect families in the top 500 list (as of June 2008)


 Overview of Interconnects of the largest machines (as of June 2008)

Rank Machine No. of nodes (cores) Interconnect Details
1 RoadRunner (Opterons + Cell BE) 122400 Fat-tree Infiniband: 2-level fat-tree, each leaf switch has 180 down links and 96 up links (18 such CUs), 12 up links from each CU connected each of the 2nd level switches
2 LLNL Blue Gene/L 212992 (106496) 3D Torus
3 ANL Intrepid (BLue Gene/P) 163840 (40960) 3D Torus
4 Ranger (SUN cluster) 62976 (3936) Fat-tree Infiniband: 3-level fat-tree, 2 switches at the top level, each of which an accomodate 3456 nodes, the different levels of the fat-tree are: chassis, the line card, the backplane
5 ORNL Jaguar (XT4) 30976 (7744) 3D Torus Cray SeaStar Interconnect: 21x16x24, mesh in the X direction and torus in the Y and Z dimension
6 FZJ Jugene (Blue Gene/P) 65536 (16384) 3D Torus
7 NMCAC Encanto (SGI Altix) 14336 (3584) Fat-tree (?)
8 CRL Eka (HP Cluster) 14384
9 IDRIS Blue Gene/P 40960 (10240) 3D Torus
10 Total Exploration Production's SGI Altix 10240 (2560) Fat-tree (?)
11 Sweden's Xeon Cluster 13728
12 Sandia Red Storm (XT3) 26569 3D Mesh Dimensions - 27x20x24
13 EDF Blue Gene/P 32768 (8192) 3D Torus
14 Watson Blue Gene/L 40960 (20480) 3D Torus
15 LBNL Franklin (XT4) 19320 3D Torus
16
17

Details about some interesting interconnects can be found below.

 Rank 1: RoadRunner

RoadRunner is a 2-level fat-tree. Each leaf switch has 180+12 down links and 96 up links and is called a control unit (CU). There are 18 CUs in total. Each CU is connected to 180 Triblades (2 Opterons and 2 Cell BEs). 12 up links from each CU are connected to each of the 2nd level switches. There are 8 2nd-level switches and have 288 ports each (180 + 12 + 96). 72 ports are not used and can be used to expand the machine.


 Rank 4: Ranger

Ranger has a 3-level Infiniband Fat-tree interconnect. This leads to a maximum of 7 hops. There are two switches at the top level, each of which can accomodate 3456 nodes. But the 3936 nodes are distributed evenly between the two switches. Each leaf switch is connected to a chassis which supports 12 blades (nodes?). The other levels of tree are within the big switches (each chassis connects to a line card which connect to the "back" of the line card and then to the backplane).

Back to Home