A fabric can have ample nominal bandwidth and still deliver inconsistent application performance. Replacing a switch with a faster one may help a genuine capacity bottleneck, but it will not automatically correct a mismatched transport configuration or a poorly understood traffic pattern.
RoCE should be evaluated as an end-to-end behavior, not as a feature enabled on one device.
Start with the path the application actually uses
Confirm the server, adapter, driver and application configuration. Check which interface and transport carry the workload. A supported network-adapter family does not prove that the selected server and software combination is qualified; versioned platform matrices define a narrower scope. R10
Then map the path through the fabric. Identify shared links and competing traffic. Average utilization may hide short periods of concentrated demand, so the observation interval and traffic definition matter.
Congestion treatment must be coherent
Traffic classification, queue mapping and congestion response need to agree across the relevant hosts and switches. A configuration copied from another platform or release may not provide the intended behavior.
NVIDIA’s Cumulus Linux 5.6 documentation distinguishes lossy and lossless RoCE profiles. Cisco’s Nexus QoS documentation also describes platform- and release-specific PFC conditions. These are reasons to follow a reviewed deployment design—not to turn PFC on universally as a substitute for one. R08R24
The acceptance record should show the policy that was applied and the observations made under the intended load.
Separate competing explanations
Suppose an illustrative job slows down when additional nodes are introduced. Several explanations remain possible: more traffic crosses a shared path, endpoint placement changes, an intended transport is not active, or congestion behavior becomes significant. The observation alone does not identify the switch as the cause.
Test one hypothesis at a time. Keep software and workload conditions stable while changing a controlled variable. Retain the results, including cases that do not support the initial theory.
| Observation | Useful next question |
|---|---|
| Slow at small scale | Is the intended host and transport path active? |
| Degrades across racks | What changes in path capacity and traffic placement? |
| Variable completion times | What concurrent traffic and congestion indicators correlate? |
| Faster links show little benefit | Is another resource limiting the workload? |
Use a benchmark as evidence, not a verdict
Collective tests can help characterize communication, but their reported metrics must be interpreted correctly. NVIDIA’s nccl-tests distinguishes algorithm bandwidth from bus bandwidth and requires the test context to make comparisons meaningful. R11
Follow those tests with a representative application run. Record message sizes, node placement, repetitions, versions and competing activity. A single peak number should not stand in for stability across the required workload range.
Buy more bandwidth when the evidence shows that capacity is the relevant constraint and the proposed design addresses it. Otherwise, the more effective upgrade may be a corrected configuration, a different traffic layout or better operational visibility. The diagnosis should determine the hardware change—not the other way around.
Engineering illustrations
These figures illustrate the design concepts discussed here. They are not measured results, certified configurations or a supplied-equipment list.
Related reading
- Nexus Leaf Design: Match Server Interfaces to Fabric Requirements
- Nexus 9300 Leaf Selection Checklist: NICs, Media, Modes and Fabric Services