Green link indicators answer a narrow question: the interfaces have established a connection. They do not show whether a distributed job uses the intended transport, whether traffic remains balanced at scale or whether the application meets its completion-time target.
An AI network should be accepted through a sequence of increasingly realistic tests. Each stage should answer a different question and retain enough context for another engineer to repeat it.
First prove the physical configuration
Compare the installed device, adapter, module and cable identifiers with the approved link schedule. Record the operating rate, port mode and relevant diagnostics. Inspect error counters over a defined interval rather than relying on a single screenshot.
Keep the software baseline with these results. Versioned platform support is important because a qualified adapter family is not evidence for every operating-system, driver and firmware combination. R10
Then verify the transport path
Confirm that the application is using the intended interfaces and communication mechanism. Check connectivity in both directions and across the relevant rack boundaries. For RoCE, inspect the configured traffic classes and congestion policy at hosts and switches; NVIDIA’s documented lossy and lossless profiles illustrate why there is no universal instruction to enable PFC on every port. R08
A passing low-concurrency test should not terminate this stage. Add the traffic patterns the design must handle, including competing transfers where those are part of the production requirement.
Test collectives before claiming application performance
Collective communication introduces a different workload from a point-to-point stream. NVIDIA’s nccl-tests provides collective tests and reports distinct algorithm-bandwidth and bus-bandwidth measures. Those metrics must retain their definitions; they are not interchangeable with physical port rate. R11
Record the collective, message-size range, GPU and process placement, node count, software versions and repetition count. Compare like with like. A larger test can change multiple variables at once, so a lower result is a symptom to investigate, not immediate proof of a defective switch.
Finish at the application boundary
Run a representative job with a fixed workload definition. Measure the outcome that matters to its owner, such as completion time or a defined step-time distribution. Note storage activity, placement and competing work so that an apparent network change is not confused with another bottleneck.
An illustrative acceptance record might look like this:
| Stage | Question | Evidence |
|---|---|---|
| Physical | Is the approved path installed and stable? | Inventory, mode and error observations |
| Transport | Is traffic using the intended mechanism? | Host/fabric configuration and controlled transfer |
| Collective | Does communication behave as expected at scale? | Repeated, fully specified collective tests |
| Application | Does the service meet its agreed objective? | Representative job results and environmental context |
Test recovery without hiding disruption
Where resilience is in scope, introduce only approved faults in a controlled environment or maintenance window. Record interrupted jobs, surviving capacity and recovery actions—not just the time until interfaces return.
Do not run intrusive laboratory benchmarking against production services without an approved method. RFC 2544 defines a benchmarking framework, while RFC 6815 specifically cautions against applying such tests to production networks. R46R47
Acceptance is complete when the evidence supports the agreed requirement. It is not complete merely because every port is up or one attractive throughput number has been captured.
Engineering illustrations
These figures illustrate the design concepts discussed here. They are not measured results, certified configurations or a supplied-equipment list.
Related reading
- AI Ethernet Fabric Design: From GPU Traffic to a Qualified BOM
- Network Switch Troubleshooting: Isolate the Failure Before Changing Configuration