Site icon QATechTools

Codex Rules for QA: Test Command Guardrails Before Agent Runs

bajpaiprashant
Codex Rules for QA: Test Command Guardrails Before Agent Runs featured image

Codex rules for QA help a team control which command prefixes may run outside the sandbox, which require a human prompt, and which must be blocked. A rule file can reduce repetitive approvals, but a broad or untested rule can also grant more authority than the test workflow needs.

This tutorial builds a disposable command-policy lab for QA engineers, SDETs, automation testers, and AI testing learners. You will create harmless test commands, write narrow prefix_rule() entries, add inline positive and negative examples, inspect the strictest matching decision, exercise wrapped commands, and verify the policy with codex execpolicy check before using it in an agent run.

Current feature status: Official OpenAI documentation labels Codex rules experimental and says they may change. Freeze the installed Codex build during validation and re-run the policy suite after an upgrade.

What Codex command rules control

The current official Codex Rules documentation describes rules as controls for commands Codex requests to run outside the sandbox. A rule matches the start of a command argument list and returns one of three decisions:

If multiple rules match, Codex applies the most restrictive result: forbidden outranks prompt, and prompt outranks allow. For QA, that means a specific block can protect a dangerous variant even when a broader test prefix is allowed.

Define the QA policy contract

Given a frozen Codex build, trusted configuration layer, rule-file hash, exact command argument list, sandbox mode, and disposable workspace, the policy checker must return the expected strictest decision and no unapproved process or side effect may occur.

Record a case ID, Codex version, operating system, repository commit, rules path and hash, command as an argument array, expected matching rule, expected decision, observed decision, matching justifications, checker exit status, and independent side-effect result. Do not use a model summary as proof that a command was blocked.

Step 1: create a disposable command fixture

Use a temporary Git repository with no credentials, remotes, production data, package installation, or external services. Add a tiny test runner that accepts only synthetic cases and writes reports under tmp/rule-lab/. The examples below use stable scripts rather than arbitrary shell fragments:

node tools/qa-probe.mjs smoke
node tools/qa-probe.mjs regression
node tools/qa-probe.mjs publish
node tools/qa-probe.mjs destructive-canary

The smoke and regression cases should read frozen fixtures and emit deterministic JSON. The publish case should attempt only a local marker that represents an external write. The destructive canary should attempt a clearly named marker outside the approved report folder, never a real deletion.

Before each trial, capture Git status, the fixture hashes, process inventory, report-directory inventory, and canary state. Repeat those checks after the policy decision. This separates policy behavior from an agent’s explanation.

Step 2: choose the correct configuration layer

Create a .rules file under a rules/ directory next to an active Codex configuration layer. The user-layer example is ~/.codex/rules/default.rules. Official documentation also says project-local rules under <repo>/.codex/rules/ load only when the project .codex/ layer is trusted.

For the lab, prefer a dedicated test profile or disposable user so the rules do not affect unrelated repositories. Record whether the file came from a user, team, or trusted project layer. Restart Codex after changing the file because rules are scanned at startup.

Step 3: start with Prompt, not Allow

The first rule should preserve human review while proving the prefix is correct:

prefix_rule(
    pattern = ["node", "tools/qa-probe.mjs"],
    decision = "prompt",
    justification = "Review each disposable QA probe before it runs",
    match = [
        "node tools/qa-probe.mjs smoke",
        "node tools/qa-probe.mjs regression",
    ],
    not_match = [
        "node tools/other-script.mjs smoke",
        "npm test",
    ],
)

The pattern is an exact argument prefix, not a substring search. Each element can be a literal or a set of allowed alternatives at that argument position. Keep the prefix long enough to identify the approved runner; allowing only node would cover every Node.js script.

Step 4: block risky cases explicitly

Add a more specific forbidden rule for the synthetic destructive case:

prefix_rule(
    pattern = ["node", "tools/qa-probe.mjs", "destructive-canary"],
    decision = "forbidden",
    justification = "Use the read-only smoke or regression case instead",
    match = [
        "node tools/qa-probe.mjs destructive-canary",
    ],
    not_match = [
        "node tools/qa-probe.mjs smoke",
    ],
)

Because the strictest decision wins, this rule should block the dangerous variant even though the broader probe prefix requests a prompt. Include a recommended safe alternative in the justification so a rejection remains actionable.

Step 5: test the file without starting an agent task

Use the documented policy checker against the exact rule file and command:

codex execpolicy check --pretty \
  --rules ~/.codex/rules/default.rules \
  -- node tools/qa-probe.mjs smoke

The checker emits JSON describing the strictest decision and the matching rules, including justifications. Run it for every match and not_match example plus independent boundary cases. Save stdout and stderr separately and validate the JSON rather than taking a screenshot alone.

Case Expected Required evidence
Exact smoke prefix Prompt Matching rule and justification
Destructive canary Forbidden Strictest rule wins; no marker
Different script No lab rule match No false allow
Extra safe argument Prompt Argument array recorded
Look-alike path No lab rule match Near-match rejected

Step 6: add inline unit cases carefully

The match and not_match fields are examples Codex validates when it loads the rules. Use them as executable design documentation. Add cases for:

A passing inline list proves only that those examples match as designed. Keep an external regression matrix so a parser or shell-wrapper change cannot silently invalidate the team’s assumptions.

Step 7: challenge unions and broad prefixes

A pattern element can contain alternatives, such as allowing view or list at one position. For QA commands, unions can be useful for two read-only report modes, but they can also hide scope expansion.

pattern = [
    "node",
    "tools/qa-report.mjs",
    ["view", "list"],
]

Test every allowed alternative and at least one adjacent forbidden verb such as publish, delete, or update. Do not use a union when separate rules would make ownership, justification, and rollback easier to review.

Step 8: verify compound-command handling

Shell wrappers deserve their own suite. The official documentation says Codex can split a linear shell script made of plain words joined by operators such as &&, ||, semicolon, or a pipe. It evaluates the individual commands and uses the strictest result.

Create a harmless compound case where an allowed report command is followed by the forbidden canary. The entire request must not become allowed merely because its first command matches. Verify each segment appears in the policy evidence and the canary remains absent.

Advanced features such as redirection, substitution, environment-variable assignments, wildcards, or control flow are treated conservatively as one wrapped invocation rather than being split. Add cases for each form your real test scripts use. Prefer stable scripts with explicit arguments over complex inline shell strings; they are easier to review and match.

Step 9: graduate one command to Allow

Only after the checker matrix passes should you consider automatic approval for a deterministic read-only command. Narrow the rule to one stable script and one safe subcommand:

prefix_rule(
    pattern = ["node", "tools/qa-probe.mjs", "smoke"],
    decision = "allow",
    justification = "Read-only synthetic smoke probe in the disposable QA lab",
    match = ["node tools/qa-probe.mjs smoke"],
    not_match = [
        "node tools/qa-probe.mjs regression",
        "node tools/qa-probe-smoke.mjs",
    ],
)

Remember that a prefix also matches additional trailing arguments. The smoke runner must reject every unknown option, and the external matrix must include appended flags such as --publish. If a trailing argument can widen behavior, keep the decision at Prompt or design a smaller wrapper with a fixed contract.

Re-run the full matrix. Confirm the broader Prompt rule still covers regression and the Forbidden rule still covers the destructive canary. Then perform a single controlled Codex run and independently verify the smoke command did not modify source files, Git state, remote state, credentials, or anything outside the approved report folder.

Step 10: keep rules and sandboxing separate

A command rule controls whether Codex may request execution outside the sandbox. It does not prove the command itself is safe, correct, deterministic, or limited to the current repository. Keep sandbox permissions narrow, use synthetic data, and test filesystem and network boundaries independently.

Do not write an Allow rule merely to make a sandbox failure disappear. Diagnose why the command needs broader access, reduce the runner’s privileges, or preserve Prompt. A test command can invoke build hooks, browser downloads, package managers, cloud CLIs, or report uploaders even when its top-level name looks familiar.

Step 11: build a QA regression suite for rules

Run the policy matrix whenever the Codex version, operating system, shell, configuration layer, repository trust, rules file, test runner, or command shape changes. The report should contain:

Gate on semantic decisions, not exact wording or JSON field order. A changed justification display may be cosmetic; a destructive canary classified as Allow is release-blocking.

Step 12: review durable approvals

Official documentation says adding a command to the TUI allow list writes to the user-layer ~/.codex/rules/default.rules. With Smart approvals, Codex may also propose a prefix rule during escalation. Review the exact proposed prefix before accepting it.

After any durable approval, diff the rule file, identify the new scope, rerun near-match and compound-command cases, and record the owner and reason. Remove stale exceptions. Never accept a whole runtime, shell, package manager, or cloud CLI just to avoid one recurring prompt.

QA rollout checklist

Official source

Codex command rules are useful when QA treats them as testable policy rather than convenience settings. Begin with Prompt, block risky variants explicitly, allow only proven read-only prefixes, validate every rule with execpolicy, challenge shell wrappers, and verify side effects outside the transcript. The result is a smaller, auditable execution boundary—not automatic proof that an agent run is correct or safe.


Exit mobile version