TorchTitan update highlights shared model code for reinforcement learning
TitanRL connects training and response generation within one codebase, with asynchronous rollouts.
The October 21 PyTorch update highlights TitanRL for reinforcement learning: improving a model through rewarded trials. Its shared codebase spans pretraining, post-training and generation through vLLM. Trial responses, called rollouts, run asynchronously, while training and generation can scale separately. This gives model researchers a connected path between producing examples and learning from them.
The project setup starts with Python 3.11 or newer and a PyTorch nightly development build. Its single-GPU Llama 3 debug example runs ten steps using a bundled tokenizer, without downloading one. The real-model route instead downloads a tokenizer and selects a recipe: a Python function returning the complete configuration. That introductory debug run is distinct from setting up reinforcement learning. For RL, the documented installation script adds vLLM, Monarch and TorchStore nightlies plus a FlashAttention build matched to the GPU.
The TitanRL guide qualifies the shared-model claim: attention modules differ, although their underlying kernels match. Supported on-policy configurations, using trials from the current policy, can make trainer and generator log probabilities bitwise identical. This numerical agreement trades away throughput; performance still depends on model shape and parallelism, and inference alone can lag native vLLM. For a new experiment, researchers supply datasets, an environment defining interactions and termination, a reward rubric and a Rollouter that generates trials. The default path handles scoring, batching, training and weight transfer, while APIs and configurations remain actively changing.
