Codex Custom Code Review Rules for QA: Test Coverage Without Noise

Codex custom code review rules for QA let teams place important repository context close to the code: preserve a compatibility contract, keep customer data out of logs, or follow a safe migration path. The hard part is not writing a rule. It is proving that the rule catches the intended risk without generating noise on clean changes.

This tutorial builds a disposable repository, adds root and service-level review rules in AGENTS.md, and evaluates them with positive violations, safe counterexamples, unrelated bugs, and scope conflicts. The result is a repeatable QA scorecard—not a one-off demonstration.

What official Codex guidance establishes

OpenAI’s current Code Review documentation says /review can review a base-branch diff, uncommitted changes, or a selected commit. The dedicated reviewer reports prioritized findings without changing the working tree. Uncommitted review includes staged, unstaged, and untracked files, while the review pane can show Unstaged, Staged, Commit, Branch, and Last turn views.

OpenAI’s July 20, 2026 article says Codex Code Review can use repository rules in AGENTS.md and cite applicable guidance in a finding. The article recommends small, scoped rule sets with an explicit safe path and evaluates four properties:

• Coverage: intended violations are detected, even in busy diffs.
• Restraint: clean changes and valid exceptions avoid unnecessary findings.
• Retention: ordinary bugs are still found alongside custom rules.
• Actionability: findings identify the relevant guidance, location, priority, and safe response.

OpenAI reports 98% recovery of required custom findings in its primary internal rule-guided eval suite versus 58.3% in its baseline control. Treat that as a reported result for that suite, not a performance guarantee for your repository.

1. Create a disposable rule lab

Use a private test repository with no production credentials, customer records, deploy keys, or writable remotes. Seed a tiny API service and a logging helper:

rule-lab/
  AGENTS.md
  src/api/response.ts
  src/logging/audit.ts
  services/billing/AGENTS.md
  services/billing/receipt.ts
  tests/contracts/
  fixtures/review-cases.json

Record the repository URL or local path, branch, HEAD SHA, chosen base, merge-base SHA, staged and unstaged hashes, untracked inventory, Codex surface and version, configured review model when applicable, review scope, and prompt hash. Hash every applicable AGENTS.md file.

Before and after each review, capture git status --short, a tree hash, file hashes, and relevant process state. The official workflow says the reviewer does not change the working tree; verify that independently rather than trusting the transcript.

2. Write two consequential rules

Choose rules that encode judgment a deterministic linter cannot easily express. Avoid vague guidance such as “use best practices.” Start with a compatibility boundary and a sensitive-data boundary:

# Code Review Rules

## Public API compatibility
- Treat existing response field names in src/api/ as external contracts.
- Flag a removal or rename unless the diff preserves the old field during a documented migration.
- Safe path: keep the existing field or add a backward-compatible alias and contract test.

## Sensitive data in logs
- Flag new logging of access tokens, session identifiers, or full customer payloads.
- Safe path: log an approved correlation ID and a redacted event type instead.
- Do not flag synthetic fixture IDs under tests/fixtures/.

Each rule states the invariant, affected scope, risky behavior, safe alternative, and exception. That gives the evaluator observable acceptance criteria.

Add a nested rule only where behavior differs:

# services/billing/AGENTS.md

## Receipt compatibility
- Preserve the numeric cents field until the mobile-client migration is complete.
- Safe path: add the formatted value without removing or changing cents.

OpenAI’s AGENTS.md documentation says Codex builds an instruction chain from the project root down to the current directory, with nearer guidance later in the combined prompt. It reads at most one applicable instruction file per directory and stops at the configured combined-size limit. Your scope tests must therefore record working directory, file path, instruction hashes, and effective size.

3. Build a balanced gold set

Do not evaluate only changes that should trigger a finding. A useful suite includes at least four lanes:

Expand each lane with realistic variations: staged, unstaged, and untracked files; rename detection; nested service paths; generated files; a busy diff with ten harmless hunks; two simultaneous violations; a valid exception; incomplete migration context; and an ambiguous change that should request evidence rather than assert a defect.

Give every expected finding a stable case ID, affected line range, expected rule ID, allowed severity range, and required safe-path concept. For negative cases, define prohibited rule IDs and the reason no finding should appear.

4. Establish a no-rule baseline

Run the suite first with the custom review sections absent. Freeze the exact diff and review instruction, then record findings without trying to force wording equality. Restore the custom rules and repeat against the same commit identities.

The baseline answers two questions: which seeded issues Codex finds without local guidance, and whether adding rules improves the target checks without suppressing ordinary bug detection. Do not compare runs if the base SHA, diff, review model, working directory, or prompt changed.

Use semantic matching. A finding counts as a target hit only when it identifies the correct risk and affected code. Keyword overlap is not enough. Store the raw finding separately from the scored interpretation so a second reviewer can audit the decision.

5. Run every review scope deliberately

Open /review and exercise the documented scopes separately:

1. Review against a base branch: verify the merge base and entire branch diff.
2. Review uncommitted changes: include staged, unstaged, and untracked test files.
3. Review a commit: use one immutable seeded changeset.
4. Custom review instructions: ask for the gold-set schema without changing the rules.

Never assume the UI scope matches the intended Git object. Save independent git diff, git diff --cached, untracked inventory, commit diff, and merge-base evidence. A missed case caused by reviewing the wrong diff is a harness failure, not a model miss.

6. Score coverage and restraint

Use counts rather than impressions:

coverage = required custom findings matched / required custom findings
restraint = clean counterexamples with no custom finding / clean counterexamples
noise_rate = unsupported custom findings / all custom-rule findings
duplicate_rate = duplicate findings / all findings

Review false negatives by failure mode: rule not loaded, wrong path scope, competing guidance, busy-diff omission, unclear invariant, or reviewer miss. Review false positives by cause: rule too broad, exception missing, unsafe keyword match, nested rule leakage, or severity inflation.

Repeat each case enough times to expose unstable outcomes, but do not require identical prose or ordering. Track whether the intended finding appears, whether clean cases remain clean, and whether citations point to the correct applicable rule.

7. Score retention and actionability

Retention prevents custom guidance from creating tunnel vision. Seed ordinary defects outside the rules: a missing null check, incorrect assertion, flaky shared fixture, or boundary error. Compare their detection rate before and after the rules are added.

An actionable finding should:

• identify the affected file and code;
• explain the consequence rather than merely repeat the rule;
• cite the correct root or nested guidance;
• recommend the documented safe path;
• calibrate severity to the real impact;
• state missing evidence when the conclusion is uncertain.

Reject a finding that cites a rule from the wrong directory, invents a requirement, demands removal of a valid exception, or proposes a breaking “fix.” A true risk with an unsafe remedy is not fully actionable.

8. Test hierarchy, conflicts, and limits

Create controlled instruction conflicts. Put a repository-wide rule at the root and a narrower billing rule in the nested file. Verify that a billing change receives both applicable context while an unrelated service does not inherit billing-only guidance.

Then exercise:

• an empty nested file;
• an override file replacing an ordinary file at one level;
• working from the root versus a nested directory;
• rules near the configured combined-size limit;
• two rules competing for the same hunk;
• a stale rule referring to a completed migration;
• a moved file crossing from one rule scope into another.

If a rule is excluded by discovery or size limits, report the configuration defect separately. Do not score the reviewer as if it had seen guidance that was never loaded.

9. Keep deterministic checks in CI

OpenAI recommends reserving repository rules for consequential, non-obvious invariants and leaving formatting and other mechanical checks to CI. Test that boundary. A rule about backward compatibility should complement a contract test, not replace it. A data-logging rule should complement secret scanning and structured logging tests.

For each target risk, document the strongest available enforcement:

• contract test for wire compatibility;
• schema test for additive changes;
• secret and token scan for obvious leakage;
• lint rule for mechanical logging APIs;
• branch protection and required approvals;
• Codex review rule for contextual judgment.

If a check can become deterministic, move it into CI and simplify the review rule. This lowers noise and makes failure repeatable.

10. Define the human release gate

A reviewer should confirm the correct Git scope, applicable rule hashes, target findings, clean counterexamples, ordinary bug results, deterministic test output, and unchanged working tree. The team should revise or remove a rule that repeatedly produces noise.

Block merge when a consequential seeded violation is missed, a safe counterexample is repeatedly flagged, the wrong rule scope is cited, ordinary defect retention falls materially, a proposed remedy breaks the safe path, or deterministic checks fail. Mark ambiguous cases for human investigation instead of forcing a pass or fail.

Screenshot-friendly evidence checklist

• Root and nested AGENTS.md rules with stable rule IDs
• Base SHA, head SHA, merge base, and selected review scope
• Seeded violation and safe counterexample diffs
• No-rule baseline findings
• Rule-guided findings with correct citations
• Clean case with no custom-rule noise
• Ordinary bug retained alongside custom checks
• Instruction hierarchy and nested-scope evidence
• Coverage, restraint, retention, actionability, and noise scorecard
• Unchanged Git state, passing deterministic checks, and human merge gate

Official OpenAI references

• Codex Code Review documentation
• Custom Code Review rules for Codex
• Custom instructions with AGENTS.md
• GitHub pull request reviews with Codex
• Codex sandbox documentation

Final takeaway

Codex custom code review rules for QA work best as a measured interface between repository knowledge and an additional reviewer. Start with two consequential invariants, include explicit safe paths, test positive and negative cases, verify instruction scope, and score misses as carefully as noise. Keep deterministic enforcement and the final merge decision outside the model.