Model prototyping on Spark

NVIDIA’s tutorial runs the text-only Nemotron-3.5 Lightning model on DGX Spark for terminal conversations and application prototyping through a local API. Its prerequisites matter: a supported Spark software stack, CUDA 13, TensorRT 10.16.1.11 or newer within the 10.x series, and Python 3.10–3.12. The examples use TensorRT Edge-LLM 0.11.1. This is a development workflow for embedded-device developers, allowing them to try prompts before connecting an application.

The repository explains how model preparation relates to execution. Its supported frontend converts Hugging Face model checkpoints into ONNX for building C++ engines. An experimental alternative builds directly from checkpoints without that conversion. Both feed the same deployment runtimes. On supported targets, a published Python package avoids compiling Edge-LLM yourself; the server extra supplies the high-level Python interface and HTTP serving. A compatible CUDA and TensorRT stack remains necessary.

The command guide requires completing installation and activating the Python environment first. Then trtedge chat accepts a Hugging Face model identifier or local checkpoint; its prompt option runs one request instead of a conversation. Chat and serving build and cache engines on first use, reusing matching engines later. The serve command starts an OpenAI-compatible HTTP server. Export and quantization need their respective optional dependencies; quantization also requires a format supported by the model and target device. Command-specific help lists available arguments.