Claude Code auto-fix for QA can watch a pull request, investigate failed CI checks or review comments, and push a repair when the change appears clear. A green rerun is not enough to trust an agent-generated fix. QA still needs to prove that the repair preserves test intent, does not hide a product defect, and is safe to merge.
This tutorial uses one low-risk pull request in a sandbox repository. You will create a controlled test failure, enable Auto-fix for that PR, inspect what Claude changes, rerun the evidence independently, exercise an ambiguous review request, and verify the stop control.
What the official workflow supports
Anthropic documents Claude Code on the web as a research-preview cloud workflow. Auto-fix subscribes to GitHub activity for a pull request. When a CI check fails or a reviewer adds a comment, Claude investigates. It can push a change for a clear fix, ask the user when a request is ambiguous or architecturally significant, and record duplicate or no-action events without editing.
The Claude GitHub App is required because Auto-fix uses PR webhooks. The setting applies to one PR at a time. It can be enabled from the web session’s CI status bar, with /autofix-pr from the PR branch, from mobile, or by giving a cloud session an existing PR URL. It can be turned off from the status bar or by telling Claude to stop watching.
Important QA boundaries
- Use a disposable branch and synthetic test data.
- Start with a low-risk test-only PR.
- Keep required reviews and branch protection enabled.
- Do not provide unnecessary secrets.
- Assess workflows triggered by PR comments before enabling Auto-fix.
- Treat the agent session as an audit clue, not proof.
Anthropic warns that Claude may reply to review threads using the connected user’s account. Although labeled as Claude Code, a reply can still trigger workflows listening for issue_comment. Map those triggers before testing, especially when they deploy infrastructure or run privileged operations.
QA scenario: repair a controlled CI failure
Create a small PR in a sandbox repository with a deliberately failing test whose correct repair is known. Record the expected failure, allowed files, intended assertion, and validation commands. Do not weaken a production gate merely to manufacture the exercise.
Example acceptance statement: “Restore the expected checkout validation message by correcting the test fixture. Do not change application code, skip the assertion, add broad retries, or reduce coverage. Run the focused test and checkout regression group.”
Step 1: prepare the repository safely
Confirm the Claude GitHub App is installed. Review repository permissions, branch protection, required checks, CODEOWNERS, and comment-triggered actions. Use the narrowest cloud network setting that supports dependency installation and tests.
Commit repository instructions stating permitted directories, forbidden shortcuts, test commands, data restrictions, and stop conditions. Cloud sessions receive committed repository configuration; local-only user settings do not automatically carry over.
PR scope: tests/checkout and fixtures only.
Never edit production checkout code.
Do not skip, mute, or weaken assertions.
Do not add blanket retries or fixed delays.
Run the focused test and checkout regression group.
If evidence suggests a product defect, stop and ask.Step 2: capture the failing baseline
Before enabling Auto-fix, run the same CI job from the starting commit. Save the SHA, runner image, lockfile, command, environment, exit code, failing assertion, logs, report, and trace. Reproduce in another clean runner if practical.
The baseline distinguishes a repair from an unrelated rerun, flaky pass, cache effect, or environment change. If the failure is not reproducible, investigate flakiness before asking an agent to edit code.
Step 3: enable Auto-fix for one PR
Open the Claude Code web session for the pull request and select Auto-fix in the CI status area. Alternatively, run /autofix-pr on the PR branch. Confirm monitoring is active for the intended PR, then trigger the controlled failure.
Record the time, PR number, starting SHA, enabled state, and failed check URL. Send one event during the first trial so cause and response remain easy to audit.
Step 4: audit the response
Read the session explanation and inspect every pushed commit. Compare the diff with the approved scope and baseline.
- Did Claude identify the evidenced root cause?
- Did it edit only approved files?
- Is the business assertion still meaningful?
- Were tests skipped, muted, relaxed, or broadly caught?
- Were retries, dependencies, fixtures, snapshots, or configuration changed?
- Are secrets, internal URLs, customer data, and generated noise absent?
- Are the commit and session traceable?
Reject a repair that turns a test green by changing expected behavior to match a defect. A legitimate product failure should become a defect report, not a weakened test.
Step 5: rerun evidence independently
Check out the agent’s exact commit in a clean environment. Run the focused test, surrounding checkout regressions, and required static checks. Preserve the SHA, commands, exit codes, reports, traces, screenshots, and logs.
Compare original and repaired runs. Confirm the assertion passes for the right reason and challenge nearby negative paths. For intermittent failures, repeat using a risk-based threshold rather than reporting one pass.
Step 6: test ambiguity handling
Add an intentionally ambiguous sandbox review comment: “Make the checkout test more robust.” Official behavior says ambiguous or architecturally significant requests should be raised to the user.
Verify Claude asks for intent, boundaries, or expected behavior. It should not silently add retries, remove assertions, or refactor unrelated code. Treat an unsafe change as a failed control test.
Step 7: test duplicate, conflict, and stop behavior
Repeat a resolved comment and verify a duplicate event does not create unnecessary edits. Advance the base branch to observe the documented limitation: GitHub does not emit the webhook Auto-fix needs when that creates a merge conflict. Open the session and request a rebase explicitly, then review it.
Turn Auto-fix off and trigger another controlled event. Confirm no agent commit or reply appears. The stop test is essential before teams rely on the per-PR toggle.
Screenshot checklist
- Sandbox PR with starting SHA and controlled failed check.
- Repository instructions with scope and stop conditions.
- Auto-fix enabled for the correct PR.
- Claude session explanation linked to the event.
- Agent diff limited to approved test files.
- Independent clean rerun report and trace.
- Ambiguous comment and clarification request.
- Disabled monitoring followed by no agent change.
Evidence package for reviewers
- Ticket, PR, starting SHA, and repaired SHA
- Original failure report, logs, and trace
- Auto-fix event and session link
- Complete diff and QA notes
- Independent focused and regression results
- Ambiguity, duplicate, conflict, and stop observations
- Known gaps, security review, and rollback plan
- Named human merge decision
Common mistakes
- Equating green CI with correctness: validate the assertion and behavior.
- Skipping the baseline: a flaky rerun can look like a repair.
- Ignoring comment triggers: replies can start privileged automation.
- Allowing broad access: use least privilege and a sandbox.
- Assuming conflicts are monitored: base-branch conflict creation lacks the webhook.
- Leaving monitoring on: test and document the stop control.
References
FAQ
Does Auto-fix merge a pull request?
The documented workflow investigates events and can push clear fixes. Keep required reviews and merge authority with humans.
Can it resolve every CI failure?
No. Ambiguous work should be escalated, and base-branch merge conflicts are not automatically detected by this event workflow.
Why test review comments separately?
Agent replies can trigger repository automation that listens for issue comments.
What proves a repair is valid?
A reviewed diff plus independent focused and regression evidence for the exact commit and human approval.
Conclusion
Claude Code auto-fix for QA works best as a monitored repair assistant. Bound the PR, preserve a baseline, audit every change, rerun deterministic tests, exercise ambiguity and stop controls, and keep merge authority with a human reviewer.