Seed 1.8: benchmarks, price and rank

Its headline results

The benchmarks that count most in the rating, with its place among the models measured the same way, and where each number came from.

• BrowseComp67.6%

  Questions whose answers are hidden deep on the web and take many searches and pages to track down.

  32nd of 56. The maker's own number, from ByteDance, via LLM Stats.

• LiveCodeBench v679.5%

  Programming contest problems collected after models were trained, from LeetCode, AtCoder and Codeforces.

  22nd of 43. The maker's own number, from ByteDance, via LLM Stats.

• MMMU-Pro73.2%

  College exam questions that need a picture to answer, such as charts, diagrams, maps and chemical structures, with more answer options than the original MMMU and some questions shown only as a screenshot.

  36th of 51. The maker's own number, from ByteDance, via LLM Stats.

• CharXiv Reasoning71.4%

  Questions about real charts from scientific papers that need reasoning across the chart, not just reading off a value.

  38th of 45. The maker's own number, from ByteDance, via LLM Stats.

• ERQA58.8%

  400 questions about images of the physical world: space, motion, actions, the state of objects and views from several cameras.

  15th of 21. The maker's own number, from ByteDance, via LLM Stats.

• SWE-bench Verified72.9%

  Real bug reports and feature requests from popular open-source Python projects. The model has to change the code so the project's hidden tests pass.

  25th of 56. The maker's own number, from ByteDance, via LLM Stats.

• MMLU-Pro84.9%

  Twelve thousand multiple-choice questions across fourteen subjects, from law and medicine to engineering, with ten options each.

  22nd of 56. The maker's own number, from ByteDance, via LLM Stats.