Skip to content

Search articles, repositories, topics and pages.

Coding Agents

How to Compare Coding Agents on One Repository Task

A fair, evidence-first protocol for comparing coding agents on the same issue, repository state and verification harness.

TIAN NEWS AI

Published Updated 4 min read

Two coding agents work from the same repository snapshot and converge on one test harness that exposes a hidden edge case

Coding-agent comparisons often measure different tasks without admitting it. One agent receives repository instructions, another starts from a cleaner checkout and a human quietly fixes the third agent’s dependency error.

A useful comparison freezes the repository, issue, permissions and verifier. The result is not a universal ranking; it is evidence about how different harnesses handle one representative task.

Updated

The protocol is tool-neutral. Agent models, harnesses, pricing and sandbox capabilities change frequently; preserve the exact run configuration and date.

Tested with

OpenAI evaluation guidance · SWE-bench task model · GitHub pull-request workflow

Choose a task with a hidden edge

The task should be small enough to repeat and rich enough to require repository understanding. Good candidates include:

  • fixing a reproduced bug across two modules;
  • adding validation with a regression test;
  • migrating one deprecated API without unrelated refactoring;
  • implementing a documented command behind an existing interface.

Avoid a task already described by a public patch if contamination matters. Create a private variation or a new failing test from a real local bug.

Write acceptance criteria before any run. Include forbidden changes such as new dependencies, schema edits or public API changes.

Freeze the environment

Create one base commit and one setup image. Every run receives:

  • the same repository commit and submodules;
  • the same issue text and project instructions;
  • the same dependency cache state;
  • the same CPU, memory and wall-clock limits where practical;
  • the same filesystem and network permissions;
  • the same visible tests.

Store hidden tests outside the agent workspace. If a harness requires different plumbing, document it as part of the product being evaluated.

Key takeaway

Pass rate alone cannot explain agent quality. Preserve the patch and trajectory so reviewers can distinguish correct reasoning, accidental success, overfitting and unsafe behavior.

Define intervention rules

Human help is the easiest way to invalidate a comparison. Decide in advance what counts as an intervention.

EventSuggested rule
Agent asks a product questionGive the same prepared answer to every run
Package registry is temporarily unavailableRestart without charging an attempt
Agent requests broader permissionsRecord and deny unless the benchmark grants them
Agent loops after the time limitStop and retain the partial patch
Tool crashes before model work beginsMark infrastructure failure separately

Run more than once if randomness is part of the system. One lucky trajectory is not a reliable estimate.

Score the patch in layers

Use a verification ladder:

  1. Task completion: visible and hidden acceptance tests pass.
  2. Regression safety: the relevant broader suite still passes.
  3. Scope discipline: the diff avoids unrelated files and dependencies.
  4. Maintainability: the solution follows local conventions and uses existing abstractions.
  5. Operational behavior: the agent respected permissions, secrets and time limits.
  6. Efficiency: elapsed time, model usage, tool calls and human interventions are recorded.

Run automated checks before revealing the agent identity to the human reviewer. Blind review cannot remove every clue, but it reduces brand-driven scoring.

Preserve an evaluation bundle

Each run should produce:

text
runs/agent-a-01/
├── base-commit.txt
├── task.md
├── environment.json
├── transcript.jsonl
├── patch.diff
├── test-results/
├── usage.json
└── review.md

Redact secrets before archiving. Keep enough tool output to explain failures without collecting unrelated repository data.

  1. Select a representative issue and build visible plus hidden acceptance tests.
  2. Freeze the base commit, environment, permissions and time budget.
  3. Write intervention rules and a scoring rubric.
  4. Run each agent in a fresh workspace and retain every trajectory.
  5. Apply the patch to a clean checkout and execute the same verifier.
  6. Conduct a blinded diff review for scope, design and risk.
  7. Publish results with confidence limits and task-specific caveats.

Inspect failures, not only totals

Classify failures into categories:

  • did not reproduce the bug;
  • edited the wrong abstraction;
  • passed visible tests but failed a hidden case;
  • introduced an unrelated regression;
  • could not operate the tools or environment;
  • exceeded budget;
  • violated a permission boundary;
  • produced a plausible explanation without a valid patch.

This taxonomy points to actionable improvements. More context may help a repository-navigation failure but worsen cost. A stronger verifier may catch shallow patches without changing the model.

Report what the benchmark cannot say

One repository task does not measure unfamiliar languages, greenfield design, long-running migrations or collaboration with a human team. Even established benchmarks such as SWE-bench define a particular issue-to-patch task and verifier.

Use the result to choose the next internal experiment, not to declare a permanent market winner.

  1. OpenAI evaluation best practices — explicit objectives, representative datasets, calibrated graders and continuous evaluation.
  2. SWE-bench repository — a reproducible issue-to-patch benchmark structure for real repositories.
  3. GitHub pull requests — diffs, checks and review as a change-validation workflow.
  4. OpenAI sandbox security — isolation and credential boundaries for agent-generated code.