Codex Promptfoo evals for QA can convert repeated manual checks into a small regression suite that lives beside the application. The valuable outcome is not a large collection of generated cases. It is a reviewable baseline that exercises the same path users reach, detects a deliberately seeded defect, and produces evidence a QA engineer can explain.

This tutorial uses a synthetic support assistant with one promise: answers must stay grounded in approved policy documents and return a valid citation identifier. The same workflow applies to tool selection, classifier labels, JSON contracts, business rules, retrieval behavior, and agent task completion.

What the official guidance establishes

OpenAI’s Codex AI application evals use case recommends choosing one user-visible promise, asking Codex to inspect before editing, exercising the application path users actually hit, keeping fixtures free of secrets and customer data, and running the baseline before changing production behavior. It describes a small Promptfoo suite with configuration, cases, an optional target adapter, and a reusable local command.

The OpenAI evaluation best-practices guide adds an important QA principle: generative systems are variable, so evaluation must be task-specific, continuous, grounded in realistic distributions, and calibrated with human judgment. Automated scores are useful evidence, not a replacement for product expertise.

Codex can help create and maintain the suite. Your team still owns the behavior contract, data approval, assertion quality, thresholds, CI policy, defect classification, and release decision.

Step 1: choose one observable promise

Do not begin with “test the chatbot.” Write a promise that can fail visibly:

For every policy question in the approved fixture set, the assistant answers only from retrieved policy text, includes a valid source identifier, refuses unsupported requests, and returns the documented response shape.

Split that promise into independent checks. Validate the response schema deterministically. Check that the source identifier exists in the retrieved fixture. Score groundedness against a narrow rubric. Add a separate refusal assertion for questions the fixture cannot answer.

Step 2: prepare sanitized seed cases

Create a disposable dataset with no production conversations, credentials, customer names, private URLs, or personal data. Start with eight to twelve cases, not hundreds:

  • Three typical questions with clear supporting passages.
  • Two paraphrases that express the same intent differently.
  • Two boundary questions whose answers require exact limits or dates from the fixture.
  • One unsupported question that must be declined.
  • One prompt-injection attempt embedded in retrieved text.
  • One malformed or empty request.

Give every case a stable identifier, purpose, approved evidence, expected invariant, and owner. Stable identifiers make failures traceable between a local run, pull request, and release report.

Step 3: ask Codex for a plan before files

If the approved Promptfoo plugin is available, the official workflow identifies $promptfoo-evals and, when a provider or adapter is missing, $promptfoo-provider-setup. Otherwise, give Codex the same bounded request without assuming a plugin:

Inspect the user-facing support-answer path and existing tests.
Propose the smallest Promptfoo eval plan for grounded answers.
Name the target, sanitized seed cases, assertions, files, command,
required services, and environment variables. Do not edit production
prompts or model settings. Stop for review before implementation.

Reject the plan if it tests only a raw model prompt while users call an application endpoint with retrieval, post-processing, or tool logic. Also reject any plan that copies production data into the repository or hides required services.

Step 4: review the suite structure

A compact first pass can contain evals/promptfooconfig.yaml, evals/tests/cases.yaml, and an adapter under evals/providers/ only when the built-in target cannot call the application path. Add one documented local command such as npm run evals.

Review every generated file. Confirm the adapter uses the same authentication boundary, retrieval step, and response parsing as the user-facing flow. Ensure timeouts and errors become explicit failed cases instead of empty successes. Keep secrets in the approved environment mechanism, never in fixtures or screenshots.

Step 5: combine deterministic and judged assertions

Prefer deterministic checks wherever the contract is exact:

  • Required fields and JSON types.
  • Allowed classifier labels or tool names.
  • Source identifiers present in the supplied fixture.
  • Forbidden strings, sensitive canaries, and unsupported actions absent.
  • Business limits and response-size bounds.

Use rubric or model-assisted scoring only for genuinely semantic qualities such as groundedness or relevance. Write narrow criteria with positive and negative examples. Calibrate those scores against a small human-labeled set, and record disagreements. Avoid exact-string equality for prose unless exact wording is itself the product requirement.

Step 6: capture the baseline before changing behavior

Run the suite against the current application. Preserve the configuration commit, case IDs, target environment, command, raw results, summary, and timestamp. A baseline with failures can still be useful if those failures are classified and approved; silently adjusting assertions until everything turns green destroys the evidence.

Evidence QA question Release use
Case result Which promise failed? Trace to requirement or defect
Raw response What did the user path return? Reproduce without relying on score alone
Assertion detail Was the check deterministic or judged? Review false positives and negatives
Baseline comparison Did the candidate regress? Gate only agreed critical changes
Human decision Was the behavior acceptable? Document risk and owner

Step 7: prove the suite with seeded defects

A green suite is not trustworthy until it catches something intentional. In a disposable branch, seed one defect at a time:

  1. Remove the source identifier from an otherwise correct answer.
  2. Bypass retrieval for one case.
  3. Allow an unsupported question to receive a confident answer.
  4. Return a wrong JSON type.
  5. Pass an invalid argument to a mocked tool.

Run the suite and confirm the expected case fails for the right reason. Revert the defect and confirm recovery. Record misses as suite defects. This mutation-style check tests the detector rather than merely demonstrating execution.

Step 8: compare a candidate change

Change one variable at a time: prompt, model configuration, retrieval settings, tool schema, or application code. Run the identical case set against baseline and candidate targets. Compare critical failures first, then score movement, latency, and cost signals if your approved runner captures them.

Repeat probabilistic cases enough to expose instability, but do not invent a universal run count. Choose repetition and thresholds from observed variance, risk, traffic, and budget. Investigate improvements that come with a new critical failure rather than averaging that failure away.

Step 9: add a controlled CI gate

Once the local command is stable, run it in CI or a release checklist. Start with reporting mode. Promote only deterministic critical assertions to blocking status until semantic scores have demonstrated acceptable agreement with human reviewers.

Pin the suite configuration through your dependency-management policy, use synthetic or approved test accounts, redact logs, bound retries, and archive artifacts for a defined retention period. A network outage, provider limit, or missing secret must be distinguishable from a product regression.

Negative test matrix

  • Missing provider credentials or unavailable local service.
  • Malformed fixture and duplicate case identifier.
  • Adapter timeout, invalid JSON, and partial response.
  • Prompt injection in user input and retrieved content.
  • Sensitive canary leakage in output or logs.
  • Unsupported tool, invalid tool arguments, and tool error.
  • Baseline target unavailable while candidate is healthy.
  • Flaky judged score near the threshold.
  • Assertion that passes an intentionally seeded defect.
  • CI retry that hides the first failure.

Screenshot plan

Capture the user-visible promise, sanitized case table, Codex plan, proposed file tree, assertion review, first baseline, seeded-defect failure, recovered run, baseline-versus-candidate comparison, CI artifact, and signed human decision. Blur repository names, endpoints, keys, account details, and private fixture text.

Release checklist

  • One user-visible promise is explicit.
  • The target exercises the real application path.
  • Fixtures are sanitized and approved.
  • Exact contracts use deterministic assertions.
  • Semantic scoring is calibrated with human labels.
  • The baseline ran before production behavior changed.
  • At least one seeded defect was detected and reverted.
  • Infrastructure failures are separate from product failures.
  • CI artifacts are traceable to case IDs and commits.
  • A human owns every waiver and release decision.

Limits

Promptfoo configuration and available Codex plugins can change, so verify current documentation and your organization’s approved tooling before copying commands. An eval suite samples known risks; it does not prove that an AI system is safe, unbiased, secure, or correct for every input. Production monitoring, exploratory testing, adversarial review, privacy assessment, and human judgment remain necessary.

The practical pattern is small and disciplined: one promise, sanitized cases, plan-first Codex assistance, a real application target, a captured baseline, seeded-defect proof, controlled comparison, and a human-reviewed gate.

Official sources