NVIDIA introduces Cosmos3-H-Dreams for surgical simulation

NVIDIA’s announcement brings generative simulation into scenes learned from clinical gallbladder-removal recordings. For surgical-robotics researchers, the distinction is crucial: realistic appearance does not establish biomechanical accuracy. The company presents surgeon evaluation and coverage of the full procedure as research directions, rather than capabilities already established by this release.

Running the simulator

The repository requires Linux, a CUDA-capable NVIDIA GPU and a Python 3.12-compatible environment. Its default configuration needs over 32 GB of GPU memory. Given an initial surgical image, a compatible checkpoint and instrument actions, the server generates video in chunks. Recorded-action MP4 output additionally requires ffmpeg. Live control requires a WebXR headset browser, two tracked controllers and HTTPS access. A headset or PC displays the simulation; the GPU server performs generation.

Adapting it remains incomplete

The linked training guide sets a narrower boundary than a complete executable recipe. Teacher-model fine-tuning and evaluation are supported, but causal teacher-forcing and Self-Forcing training are not yet supported in cosmos-framework. These are stages for preparing sequential generation and learning from generated context. The current loader also lacks the adapter for the surgical control extension. The guide therefore distinguishes its reference methodology from runnable distillation instructions. Its video model predicts scenes from supplied actions; it does not choose controller commands.