An AI reviewer is good at reading a large diff without getting tired. It is also capable of inventing a bug, missing a trust boundary and recommending a locally elegant change that violates the system’s architecture.
Use it as a triage and evidence tool. Do not make it the final approver.
Updated
The workflow describes a provider-neutral review system. Repository integrations, model behavior and data-retention controls change; verify the deployed service and organization policy.
Tested with
GitHub pull-request reviews · OpenAI guardrails and human-review guidance · NIST SSDF 1.1
Define what the reviewer should detect:
- correctness defects visible from the diff and nearby code;
- missing tests for changed behavior;
- security-sensitive data flows and permission changes;
- concurrency, transaction and error-handling risks;
- violations of explicit repository conventions;
- documentation or migration work required by the patch.
Also define what it cannot approve: product behavior, threat acceptance, schema migration safety, accessibility judgment and release readiness still need the responsible owner.
Before sending code to any external service, answer:
- Which repositories and paths are allowed?
- Are secrets and generated credentials excluded?
- Can the provider retain inputs or use them for training?
- Is customer or production data present in fixtures and logs?
- Which organization and regional settings apply?
- How are prompts, findings and user decisions audited?
Prefer a server-side integration with explicit repository permissions over a developer pasting arbitrary files into a personal chat.
Warning
A redaction regex is not a complete data-loss control. Use least-privilege access, secret scanning, path allowlists and a policy that prohibits sending restricted code or data.
The first pass is structural. Ask the model to summarize the intended change, list touched trust boundaries and identify which tests should prove the behavior. If that summary is wrong, more detailed findings will be unreliable.
The second pass is adversarial. Give the reviewer the project invariants and ask for a short list of findings, each with:
- exact file and line;
- failing scenario;
- why existing checks do not catch it;
- severity and confidence;
- smallest reasonable fix;
- a test that would fail before the fix.
Reject style opinions unless they match a written convention. Ten speculative comments create more work than one reproducible defect.
Key takeaway
An AI finding is a hypothesis until a person reproduces it or connects it to a violated invariant. Require evidence before turning generated review text into engineering work.
Use a simple policy:
| Finding | Required action |
|---|---|
| Formatting or local convention | Linter or author decision |
| Missing edge-case test | Author adds test or documents why not |
| Logic defect with reproduction | Block until fixed or explicitly accepted |
| Auth, secrets, payment or destructive operation | Security or domain-owner review |
| Architecture change | Owning maintainer approval |
| Low-confidence speculation | Do not block; optionally investigate |
Never let the model dismiss a required check or approve its own generated patch.
Run formatters, linters, type checks, tests, dependency scans and policy checks independently. Their raw outputs are evidence the AI can summarize, not decisions the AI can rewrite.
Branch protection should still require the configured human approvals and checks. An AI review can add a status or comment, but it should not hold a reusable token that bypasses those protections.
- Classify repository data and grant the reviewer least-privilege read access.
- Write the review contract and repository invariants.
- Run deterministic checks and attach their unmodified results.
- Ask for a structural summary before detailed findings.
- Require line-level evidence, a failing scenario and a proposed test.
- Have the author disposition every blocking finding.
- Route sensitive changes to the accountable owner and retain human merge authority.
- Audit accepted, rejected and escaped findings to improve the rubric.
Maintain a small set of past pull requests with known defects and known false positives. Measure:
- recall on high-severity issues;
- precision of blocking findings;
- duplicate and non-actionable comment rate;
- reviewer time saved or added;
- defects that escaped both AI and human review;
- sensitive-data or permission-policy incidents.
Re-run this set when the model, prompt, repository conventions or integration changes. A reviewer that was useful last quarter may become noisier after a product update.
If the AI service is unavailable, ordinary review must still work. If its review is delayed, the team needs a clear rule about whether the result is advisory or required. If a provider incident affects data handling, administrators need a way to disable the integration and revoke its credentials.
The best small-team workflow is intentionally boring: automated checks, concise machine triage, human judgment and an auditable merge decision.
- GitHub pull-request reviews — review decisions, protected branches and code-owner workflows.
- GitHub pull-request checks and diffs — the evidence available during review.
- OpenAI guardrails and human review — approval boundaries and independent enforcement for consequential tool actions.
- NIST Secure Software Development Framework 1.1 — organization-level secure development practices and responsibilities.
