Mission Control 2.5 adds Kubernetes recovery

NVIDIA’s release notes describe a recovery service that reads job output alongside system logs and telemetry, then distinguishes hardware trouble from software faults. It requeues the job and changes its specification to exclude nodes implicated in the latest anomaly. Support has a hardware boundary: this release does not support Run:ai or the recovery engine on Vera Rubin NVL72, despite offering early bring-up support for that platform.

The announcement explains why this matters during training: after a hardware fault, workload recovery and hardware diagnosis proceed in parallel. The scheduler can restart or reschedule the job while the hardware service isolates the suspect node. These are NVIDIA’s descriptions of its system.

The recovery guide also explains what automation cannot remove. Detecting a failure, shutting down a job and loading a saved checkpoint still take time; work performed since that saved state must be repeated. The guide describes the service as primarily designed and tested for PyTorch training. It lists TensorFlow, JAX and Ray as untested, warning that unrecognized log patterns can undermine detection and attribution and require more manual intervention. Its setup instructions cover Slurm jobs, so they should not be treated as Kubernetes enrollment instructions.