The terms that keep showing up in the cards, and what changes once you know them.
28 terms
A fixed set of tasks every model is run against under the same rules, so scores can be compared across models.
Every -Bench in a paper title is one of these. Two numbers from two different suites are not the same measurement, however alike they look.
See also Leaderboard · Data contamination · Benchmark saturation · HarnessShowed up in Releases
A model stating fabricated facts, citations, or APIs with the same confidence it uses for correct ones.
The danger is that it reads fine. Obvious errors get rejected; plausible ones ship into code and docs unchecked.
See also Calibration · Execution-based evaluation · Eval · Golden setShowed up in Papers
When benchmark questions and answers already sit in the training data, so the score measures memorization rather than skill.
A high score on an old public benchmark is suspect for this first. Re-testing on fresh tasks often drops it sharply.
See also Benchmark · Held-out set · Benchmark saturation · Golden set
An agent benchmark that hands the model a terminal and scores whether the task actually ended in the required system state.
It shows up constantly in agent posts. The score is produced by the harness, tools, and retry budget as much as by the model.
See also Harness · SWE-bench · Execution-based evaluation · Benchmark
A coding benchmark built from real GitHub issues: the agent patches the repo and the project's own tests decide pass or fail.
Nearly every coding-agent launch quotes it. The 500-task Verified subset and the full 2,294-task set are different exams, so the variant decides the number.
See also Execution-based evaluation · Harness · Terminal-Bench · pass@k
The test set plus the scoring procedure a team runs to check whether a model or agent behaves as intended.
A benchmark is a public exam; an eval is the one a team writes for its own product. That is why numbers rarely transfer between companies.
See also Benchmark · Golden set · Harness · LLM-as-a-judge
A public table ranking models by their scores on one benchmark or a bundle of them.
The top entries usually differ by fractions of a point. Ask the gap size and the number of runs before reading rank as ranking.
See also Benchmark · Elo rating · Win rate · Reproducibility
A scoring rule that samples k answers per task and counts the task solved if any one of them passes the tests.
Raising k can only raise the score, so pass@1 and pass@10 are not rivals. A code-generation number quoted without its k compares to nothing.
See also Accuracy · Execution-based evaluation · SWE-bench · Benchmark
A chess-style relative rating: two models answer the same prompt, the winner gains points and the loser loses them.
The gap converts to a win probability: 100 points means winning about 64% of the time, 20 points means 53% — a coin flip in practice.
See also LMArena · Win rate · Leaderboard · Human evaluation
Using another model to grade outputs — scoring an answer or picking a winner between two — in place of a human rater.
Cheap and fast, but judges favor long answers, whichever answer came first, and their own family's style. Model-only grading bakes that in.
See also Rubric · Human evaluation · Win rate · Eval
A measure of how surprised a model is by the next token in a text; lower means it predicted that corpus better.
Quantization posts lean on it to claim almost no quality loss. It is tied to one tokenizer and one corpus, so it cannot rank two models.
See also Accuracy · Benchmark · Calibration · Hallucination
Data deliberately kept out of training so the score reflects performance on examples the model has never seen.
Once you tune against it repeatedly it stops being held out. Scores that climb while real usage does not are the usual symptom.
See also Data contamination · Baseline · Benchmark overfitting · Eval
The share of tasks a model got right — the simplest possible score, and the easiest to quote out of context.
A headline 95.3% is half a fact. Task count, attempts per task, and whether partial credit counted decide if it compares to anything.
See also pass@k · Precision and recall · Benchmark · State of the art
A 2020 multiple-choice exam spanning 57 subjects, long used as the default proxy for a model's breadth of knowledge.
Top models all sit in the 90s, so it no longer separates them. A launch leaning on this number is telling you very little.
See also Benchmark saturation · Benchmark · Data contamination · State of the art
A site where two anonymous models answer the same prompt and human voters pick the better one, aggregated into a ranking.
Human votes cover open-ended prompts no test suite can, but style and length preferences leak in, so it often disagrees with coding scores.
See also Elo rating · Win rate · Human evaluation · Leaderboard
The fraction of head-to-head comparisons in which one model's answer was chosen over another's.
Beating B 60% of the time is not a score of 60. Swap the opponent and the number moves, so win rates only compare inside one table.
See also Elo rating · LLM-as-a-judge · LMArena · Baseline
The state where leading models all cluster near the ceiling, so the benchmark can no longer tell them apart.
This is why new -Bench papers keep appearing. On an old exam a one- or two-point gap is usually noise.
See also MMLU · Benchmark · Leaderboard · Reproducibility
Training or tuning aimed at a specific test until the score rises without the underlying capability rising with it.
This is how a top-of-the-chart model still disappoints on your own work. The only fix is re-measuring on your data.
See also Data contamination · Held-out set · Golden set · Eval
An experiment that removes one component at a time and re-measures, to show which part actually produced the gain.
The first table worth reading in a paper. Without it you cannot tell whether the gain came from the new idea or from more data.
See also Baseline · Harness · Reproducibility · Eval
Having people read the outputs and score or rank them — slow and expensive, but still the reference other scoring is checked against.
It is how you check whether automatic grading is right. If a write-up hides how many raters agreed, treat that number as soft too.
See also LLM-as-a-judge · Rubric · Win rate · LMArena
How well a model's stated confidence matches how often it is actually right — 80% sure should mean right 80% of the time.
Good calibration makes escalate-to-a-human-when-unsure a workable rule. Bad calibration makes that branch useless.
See also Hallucination · Accuracy · Precision and recall · Eval
The share of flagged items that were right, paired with the share of real ones that were caught; F1 folds both into one number.
Call every case negative for something that happens 1% of the time and accuracy reads 99%. That is how one accuracy number hides a useless detector.
See also Accuracy · Calibration · Eval · Golden set
The reference number a new method is compared against — usually the prior approach, or the trivial one.
The size of an improvement is decided by what it was compared to. A weak baseline makes an ordinary result look large, so read that line first.
See also Ablation study · Benchmark · State of the art · Win rate
Grading by running the output — code compiles, tests pass, the system reaches the required state — instead of reading it.
No grader taste enters the score, which is why code benchmarks lean on it. The flip side: anything the tests do not check counts as a pass.
See also pass@k · SWE-bench · Terminal-Bench · LLM-as-a-judge
A small, human-verified set of inputs with agreed correct answers, kept fixed so later versions can be compared against it.
Rerunning it after every prompt change turns a claim of improvement into evidence. Without one, progress and wishful thinking look alike.
See also Eval · Held-out set · Benchmark overfitting · Reproducibility
Whether re-running the same evaluation under the same settings lands on the same score.
Agent evals swing by points between runs. A score with no run count may simply be the best of several attempts.
See also Harness · Leaderboard · Benchmark · Ablation study
A written scoring guide that fixes, item by item, what earns which score before any grading happens.
For LLM judging the rubric is the metric. An 8.7 with no published criteria has nothing to do with anyone else's 8.7.
See also LLM-as-a-judge · Human evaluation · Eval · Win rate
A claim that a method scores highest of anything published so far on a particular benchmark.
Without naming the benchmark and the setting it says nothing — often the claim holds on only one sub-split.
See also Benchmark · Baseline · Leaderboard · Accuracy