NVIDIA and AI4Bharat announce a 2.8-trillion-token Indic corpus

NVIDIA and AI4Bharat have announced Nemotron-pretraining-Indic-v1, a corpus of roughly 2.8 trillion tokens covering India’s 22 constitutionally recognized languages. The collaborators also plan to release the GPU-accelerated pipeline used to generate and filter the data. The scale is notable, but the corpus is largely machine-translated and automatically judged, so its value depends on provenance, validation and how users interpret the reported token total.

What the 2.8 trillion figure contains

The project report counts 2.601 trillion tokens in native scripts and another 224.6 billion in romanized form, measured with a Gemma-4 tokenizer. Twelve languages appear in both native and Latin-script transliterations; ten appear only in native script, producing 34 language-script configurations in total.

These are retained corpus tokens, not 2.8 trillion unique source sentences or a statement of how many tokens a model must train on. The pipeline starts from 15 subsets of Nemotron-CC-v2, including web, question-answering, mathematics and code collections. Translations and script variants derived from a shared source are counted in their respective outputs. The release preserves source identifiers to support provenance and source-level deduplication, but the authors say it does not perform semantic deduplication across similar passages.

Two workflows, with different safeguards

For 12 languages, Gemma-4-31B-it translates complete documents, then evaluates its own output against the source with a rubric inspired by Multidimensional Quality Metrics. Outputs scoring at least 75 out of 100 are retained. Lower-scoring documents can be repaired and reassessed; the authors report that this recovery stage added 438.4 billion accepted tokens. Some accepted native-script text is then transliterated into Latin script.

For the other ten languages, the team uses Indic Translate from Bodhan AI and a Gemma judge, retains documents above the same threshold, and skips the repair stage. Santali requires windowed translation, while Manipuri is translated into Bengali script before conversion to Meitei Mayek. These language-specific paths make the pipeline more transparent, but they also mean quality assurance is not uniform across all 22 languages.

The score of 75 is a filter derived from the project’s rubric, not a measured percentage of factual correctness. Sampled human reviews found omissions and residual English in some translations. Initial comparisons favored the two-step repair method for reducing unwanted additions and untranslated text, yet the authors still call for broader evaluation by native speakers, especially for the ten-language route.

Training gains do not isolate the corpus effect

To test the data, the team continued pretraining a Nemotron 3.5 Super VL checkpoint for 100 billion configured training tokens. Ten billion came from a mixture of synthetic and naturally occurring Indic data; the remainder used the model’s earlier mixture. The report compares checkpoints immediately before and after that training stage, with gains of 4.98 to 9.82 percentage points across five Indic knowledge and reasoning benchmarks. A separate comparison of the released Nemotron 3.5 Super VL with Nemotron 3 Super shows improvement on five of six task benchmarks; IFEval 1.5 declined by 2.04 points.

The before-and-after comparison supports the combined training recipe. It does not isolate the synthetic corpus’s contribution, because the training mixture also contains naturally written Indic text. The release-to-release comparison includes other training changes, so it cannot be treated as a measurement of this one stage. The team also stresses that translated data cannot replace naturally written local-language material or fully capture cultural context.

The corpus and pipeline were announced for release “soon,” so availability should be checked rather than assumed. When published, the most useful features may be the retained provenance and reproducible generation process. The remaining questions are equally important: full-corpus quality, broader native-speaker review, semantic duplication and the license details attached to the final data card.