Nemotron 3.5 Lightning (30B A3B): benchmarks, price and rank

Its headline results

The benchmarks that count most in the rating, with its place among the models measured the same way, and where each number came from.

• BrowseComp37.0%

  Questions whose answers are hidden deep on the web and take many searches and pages to track down.

  52nd of 56. The maker's own number, from NVIDIA, via LLM Stats.

• Humanity's Last Exam11.7%

  About 2,500 questions written by experts in more than a hundred fields to be as hard as they could make them, many needing a diagram or image.

  65th of 74. The maker's own number, from NVIDIA, via LLM Stats.

• AA-LCR52%

  Questions that need reasoning across several long documents at once, about 100,000 tokens of text in all.

  17th of 19. The maker's own number, from NVIDIA, via LLM Stats.

• SciCode32.6%

  Research coding problems written by scientists across 16 fields, from physics and chemistry to biology and materials science.

  24th of 24. The maker's own number, from NVIDIA, via LLM Stats.

• SWE-bench Verified51.6%

  Real bug reports and feature requests from popular open-source Python projects. The model has to change the code so the project's hidden tests pass.

  52nd of 56. The maker's own number, from NVIDIA, via LLM Stats.

• GPQA Diamond75.4%

  Graduate-level questions in biology, physics and chemistry, written by PhDs so that searching the web does not give the answer away.

  The maker's own number, from NVIDIA, via LLM Stats.

• MMLU-Pro81.9%

  Twelve thousand multiple-choice questions across fourteen subjects, from law and medicine to engineering, with ten options each.

  34th of 56. The maker's own number, from NVIDIA, via LLM Stats.