[Beowulf] torus versus (fat) tree topologies
patrick at myri.com
Fri Nov 12 14:51:48 PST 2004
It was good to see you at SC.
Joachim Worringen wrote:
> Yes, a feature of the NIC, plus: the torus doesn't affect the latency
> significantly. In fact, the cost of a hop in an SCI-based torus, like an
> other reasonable modern torus, is only a low number of nanoseconds (I'd
> have to look it up right now for an exact number which I can not do from
> here; I think there's also such a number in a recent publication on
> BlueGene). A hop in a router node has about the same delay (as it is has
> a similar task).
Going trough a crossbar cost about 100-150ns these days. I would expect
the cost of one hop to be roughly the same in a torus. Now, the
interesting number is the number of hops for a given number of nodes.
With a 32-ports crossbar, you have one hop for 32 nodes. For 1280 nodes,
you can build a Clos topology of diameter 5, thus 5 hops. With a 3D
Torus, you route on a hypercube, so you have one hop for 8 nodes. I let
the pleasure to compute the number of hops for a 1280 nodes 3D torus to
a volunteer :-)
So the fabric overhead will depend on the size of the machine, as always.
Another consideration is the cabling. With 3D torus, you have 6 links
going to each node. If they are on a backplane like BlueGene, that's
fine. However, if those are short fat copper cables, you are in for a
More information about the Beowulf