Cisco · dc-leaf
Nexus Leaf Design: Match Server Interfaces to Fabric Requirements
Design Nexus leaf connectivity around server NICs, media, uplink capacity, overlays and workload-specific validation.
A leaf switch connects a server interface to a wider network service. Selecting one by port count alone leaves host attachment, uplink capacity, overlay behavior and workload performance unresolved. Begin with the servers and the traffic they must exchange, then select a platform whose documented capabilities fit that requirement.
Define server attachment precisely
List server NIC models, media, required link rates and physical routes. Separate copper server access from optical or direct-attach interfaces; a rate label does not establish interchangeable media. Record whether the host is single-connected, dual-connected or using a specific supported aggregation arrangement.
For optical links, identify the implementation at both ends. The Cisco 100G portfolio includes different optical and cable options, so “100G compatible” is not a complete link specification. R25
Calculate capacity at the actual failure boundary
Consider an illustrative leaf with forty-eight 25G server links and six 100G uplinks. Nominal server-facing capacity is 1.2 Tb/s and uplink capacity is 0.6 Tb/s, producing a 2:1 access-to-uplink ratio. Losing one uplink changes that ratio to 2.4:1. These ratios do not predict application throughput; they expose an assumption that must be checked against traffic direction and concurrency.
Decide whether the resilience objective concerns one link, one spine or a complete leaf. Those failures consume different amounts of capacity and may affect different servers.
Separate transport, overlay and workload requirements
| Requirement | What must be demonstrated |
|---|---|
| IP transport | Reachability, routing stability, MTU and path distribution |
| Overlay service | Correct endpoint attachment, segmentation and control-plane behavior |
| Host connectivity | Supported NIC/port/media combination and intended redundancy |
| RoCE workload | Consistent endpoint and fabric congestion behavior under representative load |
| Operations | Observable errors, recoverable access and repeatable configuration |
VXLAN provides an encapsulation mechanism, while an EVPN-based design adds a control-plane approach for the overlay. Their presence does not establish a complete operating design. Define which platform features and scale limits are actually required. R30R31
Treat RoCE as an end-to-end design
High port bandwidth is not sufficient to establish an effective RDMA path. Where the application uses RoCE, align the endpoint configuration, queue treatment, congestion notification and applicable flow-control policy. Use guidance for the selected Nexus platform and release; do not copy settings across ASIC generations without review. R24
A conventional server network should not inherit additional lossless-network complexity merely because a switch supports it. Include only features justified by the workload.
Accept the path, not just the leaf
Start with host attachment and link diagnostics, then validate routing or overlay services. Run representative traffic with the intended concurrency, capture error and congestion indicators, and repeat relevant checks after a controlled fault. A device-level benchmark should not be presented as an application result.
The final design should explain why the selected interfaces, uplinks and software functions are necessary. Retain the server-to-port map, qualified interconnects, capacity assumptions and test evidence with the sourcing request.
Engineering illustrations
These figures illustrate the design concepts discussed here. They are not measured results, certified configurations or a supplied-equipment list.
Related reading
- Nexus 9300 Leaf Selection Checklist: NICs, Media, Modes and Fabric Services
- Why More Bandwidth Alone Will Not Fix a RoCE Fabric