Zephon keeps TorchTitan’s training data stable when GPU counts change
Zephon lets a training run resume on a different number of GPUs while preserving the same order of training examples. The TorchTitan integration aims to make experiments comparable even when available hardware changes.
In this article
Why GPU count can distort an experiment
The Zephon team has connected its data-loading system to TorchTitan to make training data reproducible when a run resumes on a different number of GPUs. This matters because conventional loaders can change sample order when the hardware allocation changes. An ablation may then appear to measure a new data mixture or filter while partly reflecting a different training sequence.
In the team’s demonstration, a one-billion-parameter model trained on 16 GPUs was checkpointed at step 150 and resumed on 8, 32 and 64 GPUs. With a standard loader, each resumed run received a different sample sequence and followed a distinct loss curve. With Zephon, the reported SHA-256 hash of every global batch matched the original run.
A second experiment trained the same model for 20 billion tokens on different GPU counts. The authors report spreads of up to 0.82 points on DCLM Core v1 and 0.64 points on FineWeb with a standard loader. Zephon reduced those spreads to 0.05 and 0.011 points. These are results reported by Zephon’s developers, not an independent replication.
How elastic determinism works
Zephon performs tokenization, packing, shuffling and dataset mixing inside the training job instead of requiring a fully materialized processed corpus. Its architecture separates decision-making from expensive computation. Single-threaded accumulators make ordered, stateful choices, while interchangeable workers execute stateless operations in parallel and return results in sequence.
The loader also divides the global stream into fixed logical lanes rather than tying it directly to physical GPU ranks. When the number of data-parallel ranks changes, lanes are redistributed without changing their own ordered records. Checkpoints store bounded in-flight work rather than entire packing bins or shuffle buffers. During recovery, Zephon replays input from the latest flush point and discards records already consumed.
TorchTitan integration and boundaries
The TorchTitan integration replaces the default loader with a Zephon pipeline defined by a data recipe. That recipe specifies datasets, mixture weights, packing, shuffling and parallelism. Zephon state is saved beside the model checkpoint, and startup checks reject lane and batch configurations that do not divide correctly.
The guarantees have limits. Users must choose the lane count in advance, preferably as a common multiple of expected data-parallel sizes. The dataset, tokenizer, seed, mixture, context length and global batch size must remain unchanged across a resume. Zephon guarantees identical global batches, not bit-identical arithmetic: floating-point reduction order can still vary with GPU count. The accompanying paper describes competitive throughput on text and vision-language workloads, while noting that the preprint remains under revision.