Claude Code Ultrareview for QA provides a deep cloud review of a branch or pull request using multiple reviewer agents and an independent verification step. That sounds ideal for release-critical changes, but QA still needs to answer three questions: did it review the intended diff, did its findings survive independent reproduction, and did it miss any known risks?

This tutorial builds a synthetic pull request with seeded defects, compares local and ultra review modes, validates branch and PR scope, measures verified recall and false positives, exercises size and authentication boundaries, parses non-interactive JSON safely, and keeps comments, fixes, merge and release under human control.

What Ultrareview is today

Anthropic’s current Ultrareview documentation describes it as a research-preview feature. The primary interactive command is /code-review ultra; /ultrareview may be available as an alias. The feature, pricing and availability can change, so use the launch dialog and current docs as the source of truth for each trial.

The review runs on Claude Code on the web infrastructure. A fleet of agents examines the change in a remote sandbox, while a verification stage reproduces candidate findings before reporting them. Compared with local /code-review, Anthropic positions ultra review for broader, higher-signal analysis of substantial changes. That positioning is a hypothesis for QA to test, not a completeness guarantee.

Ultrareview requires a claude.ai sign-in and is unavailable for some third-party provider configurations and organizations using Zero Data Retention. When the cloud feature is unavailable, the current docs say /code-review ultra can fall back to a local review. Always verify the actual execution mode.

Define the QA review contract

Given a frozen pull-request commit with privately documented seeded defects and clean controls, review the intended diff, report only reproducible correctness issues with file locations, make no code change or external comment by default, and return enough evidence for a human reviewer to reproduce every finding.

Record the trial ID, repository, commit SHA, base branch, PR number, branch or PR mode, changed file and line counts, command text, feature availability, execution mode, account and data policy, estimated cost shown before launch, start and finish times, task or session link, finding payload hash, post setting, exit code, verified findings, misses and reviewer decision.

Step 1: create a controlled pull request

Use a private disposable repository containing a compact API or browser-test change. Seed six independent defects:

  1. An authorization check that is skipped on a retry path.
  2. A race where two parallel workers share cleanup state.
  3. A pagination boundary that drops the final record.
  4. A test that passes after swallowing the application error.
  5. A timezone comparison using local time where UTC is required.
  6. A resource cleanup omission visible only after the second test run.

Add two clean changes that look suspicious but are intentional: a validated cache shortcut and a test-only mock with explicit scope. These are false-positive controls. Keep the answer key outside the repository so reviewer agents cannot simply read it.

Every seed should have an independent deterministic reproduction and expected result. Exclude style-only concerns from the score because Ultrareview is designed to prioritize bugs.

Step 2: freeze the review target

Create a draft pull request and record the head and base SHAs. Save the exact diff, changed file count and changed line count. Ensure no automation pushes new commits during the experiment.

For branch mode, /code-review ultra compares the current branch with the default branch and includes uncommitted and staged changes. To use another base, pass its branch name. For PR mode, pass the PR number or supported PR reference. The remote sandbox clones the PR from the host rather than bundling the local working tree.

Run git status before branch mode and ensure stray edits are not accidentally uploaded. In PR mode, independently confirm the head SHA the service reviewed.

Step 3: establish a local-review baseline

Run a local /code-review against the frozen change first. Export every finding with file, line, claim, suggested impact and reproduction status. Measure duration and normal usage.

This is not a contest to prove one command superior. The baseline reveals whether ultra review adds verified coverage for your risk profile and whether the additional time, remote processing and usage-credit cost are justified.

Step 4: launch branch-mode Ultrareview

From the clean feature branch, run:

/code-review ultra

Before confirming, capture the dialog’s file and line scope, remaining included runs if shown, and estimated cost. Compare the displayed counts with your frozen diff. Cancel if they do not match.

After launch, the review runs in the background. Use /tasks to view status and the remote session link. Do not edit the reviewed branch while scoring the result. If you stop the task, Anthropic documents that the cloud session is archived and partial findings are not returned; record it as an incomplete infrastructure trial, not a clean review.

Step 5: launch PR-mode Ultrareview

Run a separate trial using the draft PR number:

/code-review ultra 1234

PR mode should review the hosted pull request, not the local working tree. Add one uncommitted local canary file after opening the PR and verify no finding references it. Conversely, add a hosted PR seed and confirm it is eligible for review.

Compare branch-mode and PR-mode root causes. Differences can come from uncommitted content, base selection, repository state, or nondeterministic analysis. Do not collapse them into a single result without explaining the scope.

Step 6: normalize and reproduce findings

Map every returned item into a fixed schema:

Field QA purpose
finding_id Stable traceability key
file and line Claim location
root_cause Normalized problem statement
seed_id Known defect or none
reproduction Independent pass/fail command and evidence
user_impact Human-assigned consequence
disposition Accepted, rejected or needs verification

Ultrareview internally verifies reported findings, but your QA process should still reproduce them in the team’s trusted environment. Remote verification may differ from production configuration, test data, permissions or external services.

Step 7: calculate useful metrics

Measure local and ultra modes across at least three identical trials:

  • Eligible-seed recall: reproduced seeds found divided by eligible seeds.
  • Precision: reproduced true findings divided by all reported findings.
  • Critical recall: high-impact seeds found divided by eligible high-impact seeds.
  • Actionability: accepted findings with a clear reproduction or verification step.
  • Clean-control accuracy: intentional controls not incorrectly flagged.
  • Variance: root causes found consistently across repeat runs.
  • Time and cost: completion duration and launch estimate or charged credits.

Count duplicated wording about one cause once. Split a single comment that contains two independently reproducible causes. Preserve raw results so normalization can be audited.

Step 8: test the research-preview boundaries

Run safe negative cases before adopting the workflow:

  • Empty diff: expect a clear nothing-to-review response.
  • Mistyped base branch: expect a branch error or suggestion, not review of an unintended target.
  • No merge base: observe the documented full-repository fallback and confirm scope before proceeding.
  • Detached checkout without refs: expect refusal rather than ambiguous analysis.
  • Oversized synthetic diff: expect refusal with the effective size limits; do not hard-code limits because Anthropic says they can change.
  • API-key-only or unsupported provider login: expect cloud unavailability and clearly identify any local fallback.
  • Zero Data Retention organization: expect Ultrareview to be unavailable.

A local fallback is not equivalent to the remote multi-agent run. Include execution_mode in every report.

Step 9: validate non-interactive output

The blocking subcommand supports scripts and CI:

claude ultrareview 1234 --json --timeout 30 --no-post

Progress and the live session URL go to standard error while findings can be printed as raw JSON to standard output. Capture the streams separately. Validate JSON syntax, required finding fields, unique IDs, allowed paths, maximum payload size and secret canaries before importing results.

The most important gate rule is exit-code semantics: the subcommand exits 0 when the review completes with or without findings. Exit 1 means launch failure, cloud error or timeout; interruption returns a distinct code and the remote review may continue. Therefore, never interpret exit 0 as “no bugs.” Parse the payload and apply your human-reviewed policy.

Step 10: test timeout and interruption

Use a short timeout in a disposable trial. Verify exit classification, standard-error message, absence of a misleading empty success report, and the remote session status. Interrupt another run and follow the printed session URL to confirm whether it continues.

Deduplicate retries with a key composed of repository, head SHA, base SHA and requested mode. A timed-out wrapper can otherwise start another paid review while the first is still running.

Step 11: keep posting disabled during validation

The current docs describe optional posting of completed findings to a GitHub PR. Default to no posting while benchmarking. Posting is an external side effect from the connected account and requires explicit per-run authorization or a deliberate non-interactive flag.

When your organization is ready to test posting, use a private disposable PR. Verify the comment author, target PR, generated-content note, one-comment behavior, failure handling and duplicate retry path. A posted comment is not a GitHub approval and must not satisfy a human review requirement.

Step 12: verify any proposed fix separately

Do not combine review and remediation in the benchmark. After a human accepts one finding, create a new branch and request the smallest fix. Inspect the diff, run the original failing reproduction, execute focused and broader regression suites, and rerun static or security checks.

Then request a fresh review against the fixed SHA. A finding can be correct while an automated patch introduces a new defect. Human approval remains the release boundary.

When to choose local or ultra review

Scenario Suggested starting mode
Small iterative test edit Local review
Authentication, payment or migration PR Ultra review plus specialists
Large diff near documented limits Split the change or use a scoped PR
Strict data residency or Zero Data Retention Approved local/CI process
Automated CI evidence Non-interactive JSON with no posting

Use Ultrareview where deeper analysis has measurable value. It should complement deterministic tests, static analysis, security tooling, ownership rules and human review—not replace them.

QA rollout checklist

  • Research-preview status and current availability are documented
  • Head SHA, base SHA, diff counts and mode are frozen
  • Seeded defects and clean controls have private answer keys
  • Branch and PR scope are tested independently
  • Actual cloud versus local fallback mode is recorded
  • Every finding is reproduced in the trusted test environment
  • Recall, precision, critical recall, variance, duration and cost are measured
  • Empty, invalid-base, no-merge-base, detached and oversized paths are exercised
  • JSON and standard-error streams are captured separately
  • Exit 0 is never treated as a no-findings gate
  • Timeouts and interruptions cannot create duplicate paid runs
  • Posting stays off until an explicit side-effect test is approved
  • Remote data handling fits organization policy
  • Suggested fixes receive independent diff and regression review
  • Human reviewers retain merge and release authority

Official sources

Deep multi-agent review becomes useful when QA tests the reviewer as carefully as the code. Freeze the scope, seed known risks, record the actual execution mode, reproduce every result, parse CI output correctly, control remote side effects and costs, and keep the final decision human-owned.