Nemotron 3 Ultra (550B A55B): benchmarks, price and rank

Its headline results

The benchmarks that count most in the rating, with its place among the models measured the same way, and where each number came from.

• BrowseComp44.4%

  Questions whose answers are hidden deep on the web and take many searches and pages to track down.

  48th of 56. The maker's own number, from NVIDIA, via LLM Stats.

• Humanity's Last Exam37.4%

  About 2,500 questions written by experts in more than a hundred fields to be as hard as they could make them, many needing a diagram or image.

  31st of 74. The maker's own number, from NVIDIA, via LLM Stats.

• LiveCodeBench v689%

  Programming contest problems collected after models were trained, from LeetCode, AtCoder and Codeforces.

  7th of 43. The maker's own number, from NVIDIA, via LLM Stats.

• AA-LCR65.4%

  Questions that need reasoning across several long documents at once, about 100,000 tokens of text in all.

  10th of 19. The maker's own number, from NVIDIA, via LLM Stats.

• IMO-AnswerBench92.3%

  Problems from the International Mathematical Olympiad, the hardest contest for school students, where the model has to reach the right final answer.

  1st of 18. The maker's own number, from NVIDIA, via LLM Stats.

• SciCode44.6%

  Research coding problems written by scientists across 16 fields, from physics and chemistry to biology and materials science.

  12th of 24. The maker's own number, from NVIDIA, via LLM Stats.

• Finance Agent v237.5%

  Real financial analysis work: finding figures in company filings and documents, multi-step calculations and written analysis.

  19th of 23. The maker's own number, from NVIDIA, via LLM Stats.

• SWE-bench Verified70.7%

  Real bug reports and feature requests from popular open-source Python projects. The model has to change the code so the project's hidden tests pass.

  33rd of 56. The maker's own number, from NVIDIA, via LLM Stats.

• GPQA Diamond87%

  Graduate-level questions in biology, physics and chemistry, written by PhDs so that searching the web does not give the answer away.

  The maker's own number, from NVIDIA, via LLM Stats.

• MMLU-Pro86.8%

  Twelve thousand multiple-choice questions across fourteen subjects, from law and medicine to engineering, with ten options each.

  8th of 56. The maker's own number, from NVIDIA, via LLM Stats.