Gemini CLI headless mode for QA can turn a one-off AI conversation into a repeatable CI step—but only if you test it like an external service. The useful contract is not merely whether the answer sounds sensible. It includes process exit status, JSON shape, streamed event order, tool-policy behavior, evidence retention, and a human decision at the end.

This tutorial builds a safe test-failure triage job around a small synthetic fixture. It deliberately separates deterministic test results from AI analysis: the test runner remains the source of pass or fail, while Gemini CLI produces a structured explanation that a QA engineer reviews.

What the official Gemini CLI behavior gives us

Google’s headless mode reference says a non-interactive run is triggered in a non-TTY environment or with -p/--prompt. The --output-format json form returns one object with response, stats, and optional error. Streaming JSON returns newline-delimited events such as init, message, tool_use, tool_result, error, and result.

The same reference documents exit codes 0 for success, 1 for a general or API failure, 42 for invalid input, and 53 when the turn limit is exceeded. These are testable interface promises. Do not reduce every nonzero result to a generic red build.

The official automation tutorial positions headless mode for CI/CD, batch processing, and wrapper tools. The policy engine reference adds an important QA control: rules can allow, deny, or request approval, while ask_user is treated as denial in non-interactive mode. That behavior should be part of the test matrix.

Define a narrow, safe use case

Create a disposable repository with a tiny calculator, one intentionally failing test, and a redacted test log. The CI job sends only the test name, sanitized stack trace, relevant diff, and a fixed instruction. It asks for a concise hypothesis, supporting evidence, uncertainty, and the next deterministic check. It must not ask Gemini CLI to merge code, publish artifacts, access production, or silently change files.

Keep secrets, customer data, tokens, internal URLs, and complete environment dumps out of the fixture. Use invented paths and values so screenshots can be published safely.

Step 1: write the output contract before the prompt

Your wrapper should validate two layers. First, validate the Gemini CLI envelope: confirm the process exit code and parse the documented output format. Second, validate your own response contract inside response. A practical triage contract can require:

  • failure_summary: one short factual statement;
  • evidence: an array of log or diff observations;
  • hypotheses: ranked possibilities, not asserted facts;
  • next_checks: deterministic commands or inspections;
  • confidence: a bounded value your pipeline treats as advisory;
  • needs_human_review: always true for release-affecting triage.

Do not treat valid JSON as correct analysis. A syntactically perfect hallucination still fails semantic review.

Step 2: use a fixed prompt and bounded context

You are assisting with QA triage. Analyze only the supplied synthetic test log and diff. Do not modify files or run tools. Distinguish observations from hypotheses. Return the requested triage fields. If evidence is insufficient, say so. A human owns the defect and release decision.

Version the prompt alongside the schema and fixtures. Record a prompt hash in the CI artifact. This makes regressions explainable when wording changes, without claiming that model prose will be byte-for-byte deterministic.

Step 3: test single-object JSON output

Run the fixed prompt with JSON output in an isolated branch or throwaway workspace. Capture standard output, standard error, duration, exit status, and the raw response as separate artifacts. Parse standard output strictly. Reject extra leading text, malformed JSON, missing response, a wrongly typed stats value, or an unexpected nonempty error on a nominal run.

Then parse the response payload into your triage schema. Assert required keys, types, maximum lengths, permitted enum values, and absence of secrets. Store validation failures as test evidence rather than trying to repair them silently.

Step 4: test the streaming JSONL lifecycle

JSONL is useful when QA needs an audit trail of the agent loop. Read one line at a time, parse each line independently, and preserve its sequence number. A nominal trace should initialize before its final result. Tool requests should have corresponding outcomes when tools are permitted. Errors can be nonfatal events, so the final process exit code and result event still need separate assertions.

Add fault-oriented parser tests: blank line, truncated final line, duplicate result, event after result, unknown event type, tool result without a tool request, very large message chunk, and invalid UTF-8 replacement characters. Your consumer should fail closed or quarantine the artifact; it should never reinterpret broken JSONL as trustworthy triage.

Step 5: map exit codes to QA outcomes

Observed exit Pipeline classification QA response
0 Headless execution completed Continue schema and semantic validation
1 General/API failure Retry only under a bounded policy; retain diagnostics
42 Input error Fail the wrapper test and fix invocation or prompt input
53 Turn limit exceeded Mark triage incomplete; reduce scope or investigate looping
Other Unknown interface result Quarantine and review; do not map to success

A zero exit status is necessary but insufficient. The response may still violate the schema, omit evidence, expose sensitive data, or contradict the test log.

Step 6: exercise policy behavior in headless mode

The official tools reference explains that tool requests are evaluated against security policy. Build a non-interactive test where a prompt attempts a forbidden write or shell action in the disposable project. Verify that the policy prevents it and that no file, process, network call, or external record changes.

Also test an ask_user rule. In headless mode it should not hang waiting for a person; the documentation says it is treated as denial. Capture the policy outcome, prove the target remained unchanged, and set a short CI timeout as a second line of defense.

Keep policies at supported user or administrator locations. The current policy reference warns that workspace-tier policy files are non-functional, so do not base a safety claim on a project-local rule that the CLI will ignore. Recheck this official note before future publication or implementation because it may change.

Step 7: add semantic QA checks

Use a gold set of 10–20 small failures with known evidence: assertion mismatch, missing fixture, timeout symptom, bad selector, environment mismatch, and intentionally ambiguous log. Score whether the response cites supplied evidence, labels uncertainty, avoids invented filenames, proposes executable next checks, and declines to diagnose when context is insufficient.

Track precision-like measures for supported claims, unsupported-claim count, schema pass rate, secret-leak rate, correct exit classification, and human acceptance rate. Compare distributions across prompt changes. Do not gate a release on stylistic similarity to one preferred answer.

Step 8: test negative and recovery paths

  1. Empty prompt or invalid argument: expect the documented input-error path.
  2. Malformed or oversized fixture: expect a controlled failure with no partial artifact promoted.
  3. Simulated API outage: ensure bounded retries and a visible inconclusive status.
  4. Turn-limit fixture: classify as incomplete, never as successful triage.
  5. Forbidden tool request: verify policy denial and unchanged workspace.
  6. Prompt-injection string inside a test log: verify it is treated as data, not instruction.
  7. Secret canary in excluded input: assert it never appears in output or logs.
  8. Rerun the same fixture: compare schema and evidence grounding, not exact prose.

Screenshot plan for a reproducible tutorial

Capture the synthetic repository tree, redacted failing log, fixed prompt, JSON envelope, schema validator result, JSONL sequence, exit-code matrix, denied tool attempt, semantic scorecard, and final human-review checkpoint. Blur usernames and machine paths. Never include credentials or real customer failures.

Release checklist

  • The deterministic test runner remains the pass/fail authority.
  • Inputs are synthetic or redacted and limited to the failure.
  • Prompt and schema are versioned.
  • JSON or JSONL is parsed strictly.
  • Every documented exit code has an explicit branch.
  • Policy tests prove prohibited actions leave no side effects.
  • Retries and timeouts are bounded.
  • Raw and validated artifacts have appropriate retention controls.
  • Unsupported claims are measured.
  • A QA engineer reviews triage before defect, code, or release action.

Limits to keep visible

Gemini CLI can summarize evidence and suggest useful next checks, but generative output can be incomplete or wrong. Output-format validation cannot prove factual correctness. Policy configuration also needs its own tests, and documentation can change. Re-review the official headless, automation, policy, and tools pages when updating the workflow.

The safest pattern is simple: deterministic tests produce evidence, a constrained headless call structures that evidence, automated checks validate the interface, and a human QA engineer owns the conclusion.

Official sources