NVIDIA describes automated recovery for training jobs

NVIDIA describes a service that coordinates recovery when distributed training stalls or fails. It uses logs and scheduler signals to identify interruptions, then applies policy to isolate implicated nodes and restart or stop workloads. New deployments initially record recommendations without scheduler actions. Operators compare that advice with their engineers’ decisions before enabling action. Restarted applications load their last checkpoint, a saved training state, through their existing restart logic.

The repository places important boundaries around adoption: the project is experimental, not ready for production, and its interfaces and configuration may change. It supports the Slurm workload scheduler, including Slurm running on Kubernetes through Slinky; native Kubernetes workload recovery is not implemented. Operators receive source and charts and build their own images. Restarting an allocation, restoring useful training progress and finishing training are separate milestones. The service manages workloads; it does not repair hardware or store checkpoints.

The diagnosis guide adds an access prerequisite. NVIDIA Attribution, the component that examines workload and node evidence, is enabled by default and requires authorized upstream source access. A failed source fetch does not automatically disable it. To build without that source, operators must explicitly select OFF. They must apply the disabled configuration after the installation values for the corresponding degraded release. This runs recovery without provider diagnosis and produces no attributions; it does not validate the diagnostic component. Healthy service instances alone are not diagnostic evidence.