Skip to content

Search articles, repositories, topics and pages.

Model Evaluation

Model Evaluation in Practice

How to compare two models on your own tasks with a small dataset, simple graders and a report you can actually read.

TIAN NEWS AI

Published Updated 2 min read

A dataset feeding a runner that calls two models, with graders scoring the outputs into a report

The question

Public benchmarks tell you how models compare on someone else's tasks. The question that matters for a product is which of two models is better at yours. This walkthrough uses one example task throughout: turning messy bug reports into structured issues.

The setup

  1. Dataset. A few dozen bug reports from your own tracker (the example uses 60), each with a hand-written expected issue title, component and severity.
  2. Runner. The same prompt for both models, temperature 0, three runs per example.
  3. Graders. Exact match for component and severity, and an LLM judge for the title.
  4. Report. One table per field, plus the ten examples where the models disagreed most.

Illustrative numbers

The example is illustrative: no dataset, model run or raw output stands behind it. The results table shows how to report this kind of comparison, not measured data, so never read it as evidence about any model. In this setup each model produces 180 outputs: 60 reports multiplied by three runs.

grade.py, python
def grade(expected: dict, actual: dict) -> dict:
    return {
        "component": expected["component"] == actual.get("component"),
        "severity": expected["severity"] == actual.get("severity"),
    }

Exact-match graders are boring and reliable. Use them wherever the answer has a fixed set of values, and keep LLM judges for free text.

Results

FieldModel AModel B
Component (exact)88%81%
Severity (exact)74%79%
Title (judge, 1–5)4.14.3

In this illustration neither model wins everywhere: Model A is better at picking the component, Model B at severity and titles. That split is the typical outcome worth planning for.

Small datasets are noisy

With only 60 unique reports and three correlated runs per report, a difference of a few percentage points is not enough to claim a stable winner without confidence intervals or a paired significance analysis. Look at the disagreements before drawing conclusions.

What to look for

  • Read the disagreements before the averages. Some "wrong" answers turn out to be labelling mistakes in the dataset, and fixing them changes the comparison.
  • Run each example more than once. Repeated runs expose unstable answers, even at temperature 0.
  • Prefer a small dataset from real work over a larger synthetic set that does not resemble your inputs.
  • LLM judges need their own checks. Blind model identity, vary answer order and manually review a sample to catch position bias or an unreliable rubric.

Key takeaway

Evaluate on your own tasks, grade what you can grade exactly, and read the examples where models disagree.

  1. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — evidence and limitations for model-based judging.