Exploiting communication and packaging locality for cost-effective large scale networks.

ABSTRACTAs processing power increases, maintaining the balance between network and computing is becoming increasingly difficult. The two major contributors to this imbalance are the cost and the power of high bandwidth networks, and both network cost and power are heavily impacted by the type of signaling used. Reducing the length of a network link leads to both lower cost and lower power. Unfortunately, the low dimension mesh and torus topologies that enable the shortest physical links also scale poorly in terms of hop count and global bandwidth. In contrast, topologies with low hop count and high global bandwidth have a large fraction of physical links that are several meters long. We propose the cube collective topology --- a hierarchical topology that uses a mesh topology locally to minimize link length and an all-to-all topology globally to minimize global hops. The result is that over 80% of the links can be very short (under 1 meter). This enables significant reductions in both network cost and network power, while still providing a balance of high global and high local bandwidth.