PyTorch 2.15 changes the default for NVIDIA distributed training

PyTorch’s release announcement describes a change for teams training models across NVIDIA GPUs: the software that exchanges data between participating GPUs has been replaced as the default. The rewritten NCCL backend was an optional preview in the preceding release; it now serves ordinary nccl process groups. The announcement still marks this interface as unstable.

The implementation change retains a way back: setting TORCH_DIST_USE_NCCL2=0 or choosing the nccl-legacy backend selects the previous implementation. Leaving the variable unset, or setting it to 1, selects the replacement. The change also addresses compatibility gaps involving fallback GPU selection, the exception raised for unsupported bitwise reductions, and timeout messages. These are concrete compatibility adjustments, rather than a promise that every existing workload behaves identically.

A diagnostics change addresses stalled work. Previously, NCCL2 exposed a trace-dump endpoint but did not publish its watchdog health to the debug server. The debug frontend can now request recorder traces from every participating process when any process becomes unhealthy. A fatal abort receives a configurable grace period for that server, and failures during blocking waits enter the same reporting path.

The developers include an example that deliberately stalls a collective operation, disables local fallback dumping, and checks whether debug-server polling produces a trace for each process. It illustrates the diagnostic path; it is not an independent performance benchmark.