Models & AIPractitioner
AI Benchmark
Standardised tests that measure what a model can actually do.
An AI benchmark is a standardised evaluation — a fixed test set with known correct answers — that allows objective comparison between models.
Key benchmarks in 2026:
| Benchmark | What it measures |
|---|---|
| MMLU | Broad knowledge across 57 subjects — medicine, law, maths, history |
| GPQA | Graduate-level science questions only PhD experts can reliably answer |
| SWE-bench | Real GitHub issues — can the model write code that fixes the bug? |
| AIME | American Invitational Math Exam — hard competition maths |
| HumanEval | Code generation correctness across standard programming tasks |
| HELM | Holistic multi-task evaluation across safety + capability dimensions |
| ARC-AGI | Abstract visual reasoning tasks humans find easy, models find hard |
Caveats:
- Contamination — if benchmark questions appeared in training data, scores are inflated
- Saturation — top models now score 85–90% on MMLU; it no longer differentiates them
- Task gap — scoring 90% on a benchmark ≠ 90% reliability on your specific use case
- Benchmark gaming — labs optimise for popular benchmarks; this doesn't always generalise
Rule of thumb: Treat benchmarks as a rough filter. Then build your own eval on tasks that mirror production.
In plain terms
A university entrance exam: useful for ranking applicants, but a 99th-percentile scorer might still be terrible at the specific job you're hiring for.