Designing a 400G data center requires careful planning of network topology, cabling infrastructure, power consumption, and cooling. This guide walks through the key decisions for a high-density 400G deployment.
The spine-leaf topology is the standard for modern data centers:
Spine Layer
[N8560-32CQ x 4-8]
| |
+--------------+ +--------------+
| |
Leaf-01 Leaf-02
[S5860-48S6Q] [S5860-48S6Q]
| |
Servers Servers
(48 x 10G/25G) (48 x 10G/25G)
- Predictable latency: Every server-to-server hop is 2 hops
- Scalability: Add spines or leaves without redesign
- High bandwidth: 400G spines provide massive oversubscription ratio
- Failure isolation: Leaf failures don't propagate
| Switch | 400G Ports | 100G Ports | Total 400G Bandwidth |
|---|
| N8560-32CQ | 32 | 32 (breakout) | 10.24 Tbps |
| N8550-48V6Q | 48 | - | 19.2 Tbps |
| Role | Recommended Ratio | Example |
|---|
| Compute (leaf-to-spine) | 3:1 to 4:1 | 48x25G server / 4x400G uplink |
| Storage (leaf-to-spine) | 1:1 or 2:1 | 8x100G storage / 4x400G uplink |
| AI/HPC | 1:1 | 8x200G InfiniBand / 4x400G uplink |
For 400G short-reach (up to 100m on OM4), MPO-16 connectors are standard:
# MPO-16 fiber pinout (8 TX + 8 RX)
# 400G-SR8 uses all 16 fibers
- 8 transmit lanes (parallel)
- 8 receive lanes (parallel)
- Total: 16-fiber MPO-16 connector
Use MPO-16 to 4x MPO-12 fanouts for 100G breakout at leaf switches:
| From Spine | Fanout | To Leaf Ports |
|---|
| 400G MPO-16 | 4x MPO-12 fanout | 4x QSFP28 @ 100G |
| 400G MPO-16 | Direct | 1x QSFP-DD @ 400G |
| Distance | Transceiver | Cable | Type |
|---|
| <500m | QSFP-DD-400G-DR4 | SMF MPO-12 | Single-mode |
| <2km | QSFP-DD-400G-FR4 | SMF LC | Single-mode duplex |
| 10km+ | QSFP-DD-400G-LR4 | SMF LC | Single-mode |
| Distance | Transceiver | Cable | Type |
|---|
| <100m | QSFP-DD-400G-SR8 | MMF MPO-16 | Multimode |
| <500m | QSFP-DD-400G-DR4 | SMF MPO-12 | Single-mode |
| Component | Power (W) | Heat (BTU/hr) |
|---|
| N8560-32CQ (32x400G) | 650W | 2,218 |
| N8550-48V6Q (48x400G) | 900W | 3,071 |
| QSFP-DD-400G-SR8 | 10W each | 34 each |
| S5860-48S6Q (leaf) | 180W | 614 |
Leaf Switch (S5860-48S6Q): 180W
Server uplinks (48x 25G):
- 16x SFP-25G-SR: 32W
Transceivers (leaf-to-spine):
- 4x QSFP-100G-SR4: 14W
-----------------------------------------
Total per rack switch: ~226W
Spine Switch (N8560-32CQ, 32 ports):
- 32x QSFP-DD-400G-SR8: 320W
- Switch ASIC + fans: 330W
-----------------------------------------
Total per spine: ~650W
- Front-to-back airflow: Standard for most 400G switches
- Cabinet depth: Minimum 1000mm for cable management
- Temperature: 18-27C (64.4-80.6F) inlet recommended
- Humidity: 40-60% RH non-condensing
| Link Type | Distance | Latency (one-way) |
|---|
| Server to Leaf (10m) | 10m | < 0.1 us |
| Leaf to Spine (50m) | 50m | < 0.25 us |
| Total leaf-to-leaf | 60m | < 0.5 us |
- Server ports: 9216 bytes (jumbo frames)
- Spine-leaf uplinks: 9216 bytes
- VXLAN encapsulated: Minimum 1550 bytes end-to-end
