Coding-agent comparisons often measure different tasks without admitting it. One agent receives repository instructions, another starts from a cleaner checkout and a human quietly fixes the third agent’s dependency error.
A useful comparison freezes the repository, issue, permissions and verifier. The result is not a universal ranking; it is evidence about how different harnesses handle one representative task.
Updated
The protocol is tool-neutral. Agent models, harnesses, pricing and sandbox capabilities change frequently; preserve the exact run configuration and date.
Tested with
OpenAI evaluation guidance · SWE-bench task model · GitHub pull-request workflow
The task should be small enough to repeat and rich enough to require repository understanding. Good candidates include:
- fixing a reproduced bug across two modules;
- adding validation with a regression test;
- migrating one deprecated API without unrelated refactoring;
- implementing a documented command behind an existing interface.
Avoid a task already described by a public patch if contamination matters. Create a private variation or a new failing test from a real local bug.
Write acceptance criteria before any run. Include forbidden changes such as new dependencies, schema edits or public API changes.
Create one base commit and one setup image. Every run receives:
- the same repository commit and submodules;
- the same issue text and project instructions;
- the same dependency cache state;
- the same CPU, memory and wall-clock limits where practical;
- the same filesystem and network permissions;
- the same visible tests.
Store hidden tests outside the agent workspace. If a harness requires different plumbing, document it as part of the product being evaluated.
Key takeaway
Pass rate alone cannot explain agent quality. Preserve the patch and trajectory so reviewers can distinguish correct reasoning, accidental success, overfitting and unsafe behavior.
Human help is the easiest way to invalidate a comparison. Decide in advance what counts as an intervention.
| Event | Suggested rule |
|---|---|
| Agent asks a product question | Give the same prepared answer to every run |
| Package registry is temporarily unavailable | Restart without charging an attempt |
| Agent requests broader permissions | Record and deny unless the benchmark grants them |
| Agent loops after the time limit | Stop and retain the partial patch |
| Tool crashes before model work begins | Mark infrastructure failure separately |
Run more than once if randomness is part of the system. One lucky trajectory is not a reliable estimate.
Use a verification ladder:
- Task completion: visible and hidden acceptance tests pass.
- Regression safety: the relevant broader suite still passes.
- Scope discipline: the diff avoids unrelated files and dependencies.
- Maintainability: the solution follows local conventions and uses existing abstractions.
- Operational behavior: the agent respected permissions, secrets and time limits.
- Efficiency: elapsed time, model usage, tool calls and human interventions are recorded.
Run automated checks before revealing the agent identity to the human reviewer. Blind review cannot remove every clue, but it reduces brand-driven scoring.
Each run should produce:
runs/agent-a-01/
├── base-commit.txt
├── task.md
├── environment.json
├── transcript.jsonl
├── patch.diff
├── test-results/
├── usage.json
└── review.mdRedact secrets before archiving. Keep enough tool output to explain failures without collecting unrelated repository data.
- Select a representative issue and build visible plus hidden acceptance tests.
- Freeze the base commit, environment, permissions and time budget.
- Write intervention rules and a scoring rubric.
- Run each agent in a fresh workspace and retain every trajectory.
- Apply the patch to a clean checkout and execute the same verifier.
- Conduct a blinded diff review for scope, design and risk.
- Publish results with confidence limits and task-specific caveats.
Classify failures into categories:
- did not reproduce the bug;
- edited the wrong abstraction;
- passed visible tests but failed a hidden case;
- introduced an unrelated regression;
- could not operate the tools or environment;
- exceeded budget;
- violated a permission boundary;
- produced a plausible explanation without a valid patch.
This taxonomy points to actionable improvements. More context may help a repository-navigation failure but worsen cost. A stronger verifier may catch shallow patches without changing the model.
One repository task does not measure unfamiliar languages, greenfield design, long-running migrations or collaboration with a human team. Even established benchmarks such as SWE-bench define a particular issue-to-patch task and verifier.
Use the result to choose the next internal experiment, not to declare a permanent market winner.
- OpenAI evaluation best practices — explicit objectives, representative datasets, calibrated graders and continuous evaluation.
- SWE-bench repository — a reproducible issue-to-patch benchmark structure for real repositories.
- GitHub pull requests — diffs, checks and review as a change-validation workflow.
- OpenAI sandbox security — isolation and credential boundaries for agent-generated code.
