StateBench tests changing information

Ai2’s announcement introduces StateBench for checking whether language models follow information as it changes across a long document, including code. Retrieving an initially assigned value differs from finding its latest replacement. Harder tasks require carrying accumulated changes forward or following dependencies between variables. Missing an update can change the final answer. These distinctions help researchers investigate which kinds of tracking a model handles. The framework currently assumes a fixed set of variables; it does not cover conversations that introduce new people or objects over time.

The implementation guide defines copying as taking another variable’s current value without a lasting connection. Validate configuration before building. The default builds all splits together. For separate runs, use a shared output directory, make sure test appears before train in the YAML split mapping, build test first, and set training’s disjoint_from: [test]. Training excludes persisted test semantic identifiers. A sample command previews the sequence, prompt and expected answer without writing datasets. Each row has a unique, generation-stable example_id; content-derived instance_id identifies semantic duplicates. The manifest counts unique semantic instances and reports semantic overlap across splits.