Benchmark
In shortAn AI benchmark is a fixed set of tasks with a scoring rule, used to compare models on the same test.
In plain words
A benchmark is a standard test for models. It has a set of questions or tasks and a rule for scoring the answers. Everyone runs the same test, so the scores can be compared. One well-known example is MMLU, a multiple-choice test with questions from 57 subjects such as elementary mathematics, US history, computer science and law. Another is HELM, a project by Stanford researchers that measures models across many scenarios and many metrics, not only accuracy but also things like robustness, fairness and efficiency.
Providers often publish benchmark tables when they release a model. Such tables rank models and make claims like "best at coding". The numbers are real, but they answer a narrow question.
Why it matters
- A starting point: Scores help you shortlist models before you test them. See choose a model.
- Narrow tests: A multiple-choice exam does not measure how well a model drafts your emails, follows your style or avoids hallucination on your documents.
- Test data can leak: If test questions end up in a model's training data, the score overstates its skill. Public benchmarks are exposed to this risk.
- Saturation: When many models score near the top, the test no longer tells them apart.
- Setup matters: The prompt, the number of tries, the use of tools and the thinking effort can change a score. Two tables from two sources may not be comparable. A reasoning model with more thinking usually scores higher on hard tests, at higher cost.
- Agents: Tests of multi-step work with tools are newer and vary a lot in setup. Compare with care (AI agent).
Example
Two models score within a point of each other on a public coding test. You need a model that rewrites support emails in your firm's tone. You take 20 real emails, write what a good answer looks like, run both models, and rate the results blind. One model clearly wins on your task, although it ranked lower on the public test. Your small test told you more than the table.
Common misconceptions
- "The top score means the best model for me." Only for that test. Check your own use case.
- "A benchmark is neutral." Someone chose the tasks, the metric and the settings. Read how the test was run and who ran it.
- "Higher is always meaningfully higher." Small gaps are often noise.
Questions and answers
What is a benchmark in AI?
A fixed set of tasks with a scoring rule, used to compare models on the same test.
Can I trust benchmark scores from providers?
They are a useful hint, but providers choose the tests and settings. Look for independent results and test the model on your own task.
How do I test a model for my own use?
Collect 10 to 30 real examples, write down what a good answer looks like, run the models you consider, and compare the results side by side.
Read next
- Large language model (LLM)
A large language model (LLM) is an AI system trained on huge amounts of text to predict the next token, which lets it write, summarise and answer.
- How to choose an AI model
Match the model to the job: small for volume, mid for daily work, top or reasoning for hard problems and local for private data, with a decision tree and a test.
- Hallucination
A hallucination is a fluent, confident AI answer that is false, invented or not backed by the sources it was given.
- Reasoning model
A reasoning model spends extra tokens on hidden step-by-step thinking before it writes its answer, which helps on hard problems but costs more.
- AI agent
An AI agent is a system where a model plans, uses tools and checks results in a loop to reach a goal with little step-by-step guidance.