Public benchmarks tell you how models compare on someone else's tasks. The question that matters for a product is which of two models is better at yours. This walkthrough uses one example task throughout: turning messy bug reports into structured issues.
- Dataset. A few dozen bug reports from your own tracker (the example uses 60), each with a hand-written expected issue title, component and severity.
- Runner. The same prompt for both models, temperature 0, three runs per example.
- Graders. Exact match for component and severity, and an LLM judge for the title.
- Report. One table per field, plus the ten examples where the models disagreed most.
Illustrative numbers
The example is illustrative: no dataset, model run or raw output stands behind it. The results table shows how to report this kind of comparison, not measured data, so never read it as evidence about any model. In this setup each model produces 180 outputs: 60 reports multiplied by three runs.
def grade(expected: dict, actual: dict) -> dict:
return {
"component": expected["component"] == actual.get("component"),
"severity": expected["severity"] == actual.get("severity"),
}Exact-match graders are boring and reliable. Use them wherever the answer has a fixed set of values, and keep LLM judges for free text.
| Field | Model A | Model B |
|---|---|---|
| Component (exact) | 88% | 81% |
| Severity (exact) | 74% | 79% |
| Title (judge, 1–5) | 4.1 | 4.3 |
In this illustration neither model wins everywhere: Model A is better at picking the component, Model B at severity and titles. That split is the typical outcome worth planning for.
Small datasets are noisy
With only 60 unique reports and three correlated runs per report, a difference of a few percentage points is not enough to claim a stable winner without confidence intervals or a paired significance analysis. Look at the disagreements before drawing conclusions.
- Read the disagreements before the averages. Some "wrong" answers turn out to be labelling mistakes in the dataset, and fixing them changes the comparison.
- Run each example more than once. Repeated runs expose unstable answers, even at temperature 0.
- Prefer a small dataset from real work over a larger synthetic set that does not resemble your inputs.
- LLM judges need their own checks. Blind model identity, vary answer order and manually review a sample to catch position bias or an unreliable rubric.
Key takeaway
Evaluate on your own tasks, grade what you can grade exactly, and read the examples where models disagree.
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — evidence and limitations for model-based judging.