NVIDIA shows how agent failures become tests

From a mistake to a test

In NVIDIA’s example, an assistant at a fictional company sends a message when asked to draft one. Developers investigate recorded steps, create tests, change instructions and compare results. The target is the agent harness: the prompts, tools and surrounding code that shape its behavior.

Investigate the recorded behavior

Compass searches stored execution records for recurring issues. Its tool checks look for invalid arguments, failures, missing results and unproductive retries; another signal compares behavior with the agent’s stated rules. Developers should inspect the supporting records before acting on a finding. This remains an early research preview, explicitly not intended for production use. Its interfaces, configuration and output formats can change incompatibly, so a finding is an investigation lead, not an established diagnosis.

Check what the tests cover

Eval Author helps a coding assistant construct evaluations from the agent repository and intended behavior. It supports Harbor and NeMo Gym formats. A first suite can begin without traces, but a complete coverage audit requires records from completed evaluation runs. Missing evidence of a behavior does not by itself prove a test is absent. Review generated cases, scoring checks and coverage findings: this research preview can produce tasks that are too easy.