Gemma 4 12B: benchmarks, price and rank

Its headline results

The benchmarks that count most in the rating, with its place among the models measured the same way, and where each number came from.

• Humanity's Last Exam5.2%

  About 2,500 questions written by experts in more than a hundred fields to be as hard as they could make them, many needing a diagram or image.

  72nd of 74. The maker's own number, from Google, via LLM Stats.

• LiveCodeBench v672%

  Programming contest problems collected after models were trained, from LeetCode, AtCoder and Codeforces.

  26th of 43. The maker's own number, from Google, via LLM Stats.

• MMMU-Pro69.1%

  College exam questions that need a picture to answer, such as charts, diagrams, maps and chemical structures, with more answer options than the original MMMU and some questions shown only as a screenshot.

  39th of 51. The maker's own number, from Google, via LLM Stats.

• MRCR v2 (8 needles)43.4%

  A very long conversation full of similar requests, where the model has to find and repeat particular earlier answers, eight of them at once.

  13th of 24. The maker's own number, from Google, via LLM Stats.

• GPQA Diamond78.8%

  Graduate-level questions in biology, physics and chemistry, written by PhDs so that searching the web does not give the answer away.

  The maker's own number, from Google, via LLM Stats.

• MMLU-Pro77.2%

  Twelve thousand multiple-choice questions across fourteen subjects, from law and medicine to engineering, with ten options each.

  48th of 56. The maker's own number, from Google, via LLM Stats.