Copilot code review effort levels for QA let teams choose between a faster standard review and deeper analysis for riskier pull requests. The useful QA question is not whether Balanced sounds more thorough. It is whether each level produces repeatable, evidence-linked findings for the changes your team actually ships.

This tutorial creates one synthetic test-automation pull request, seeds known defects, runs Lite and Balanced reviews under controlled conditions, normalizes the comments, measures detection quality and cost, tests fallback behavior, and keeps the merge decision with a human reviewer.

What Lite and Balanced mean

GitHub announced that Lite and Balanced effort levels are generally available. Lite is intended for straightforward changes. Balanced routes larger, more complex, or sensitive changes to deeper analysis with a higher-reasoning model. The selected effort applies to that individual manual review and does not alter repository or organization defaults.

The current Copilot code review documentation describes Lite as the standard default for targeted feedback and Balanced as longer analysis for complex logic, security-sensitive code, cross-service changes, or strict quality standards. Balanced uses more AI credits and may consume marginally more GitHub Actions minutes. After a review, the effort level appears in the pull-request overview comment; GitHub’s changelog also says it appears in timeline events.

Neither level is a release gate by itself. Copilot comments are hypotheses that need reproduction, deterministic tests, and human judgment.

Define the QA comparison contract

Given the same repository snapshot, pull-request diff, instructions, tools and review configuration, compare Lite and Balanced on known seeded defects. Record what each review found, missed or misstated; verify every actionable comment; and do not merge or apply suggested fixes automatically.

Use a private disposable repository with synthetic data. Record the commit SHA, base branch, PR number, repository and organization effort defaults, selected manual effort, custom-instruction hash, enabled MCP and skills context, runner type, Actions availability, review timestamp, overview label, timeline label, AI-credit usage, Actions minutes and reviewer outcome.

Step 1: build a controlled pull request

Create a small Playwright or API-test change with six seeded issues:

  1. A flaky selector that depends on presentation text.
  2. A fixed delay where a state-based wait is required.
  3. A happy-path assertion that never proves the negative outcome.
  4. Test data shared across parallel workers without isolation.
  5. An API helper that drops a required authorization header on one retry path.
  6. A cross-service cleanup omission that leaves synthetic records behind.

Add two clean changes as controls: a stable selector improvement and a correct boundary assertion. These help measure false positives. Give every seed a hidden test ID in your evaluation sheet, but do not reveal the answer key in the PR description or repository instructions.

Step 2: document excluded evidence

GitHub’s current docs say Copilot code review excludes dependency-management files such as package.json and Gemfile.lock, log files and SVG files. Do not seed the only proof of a defect in an excluded file and then count a miss against the effort level.

Still include one harmless excluded-file change to verify the boundary. Record that it was outside review coverage. If dependency or log evidence matters to the merge decision, require a separate human or tool-based check.

Step 3: freeze review context

Copilot code review can use repository custom instructions, agent instructions, agent skills and MCP context. The current GitHub how-to says these instructions and skills are read from the pull request’s head branch, so changes to review guidance inside the PR can affect the review itself.

For the first comparison, keep the context minimal and identical. Store a short repository instruction that asks for file references, user impact, reproduction steps and a suggested verification test. Disable optional MCP sources unless they are part of the experiment. Hash the instruction files and export the repository settings before each run.

This prevents a context change from being misattributed to Lite versus Balanced.

Step 4: run the Lite review

Open the pull request on GitHub. In the Reviewers section, request Copilot and select Lite. Capture the selection before submission. After completion, save screenshots of the overview comment and timeline event showing Lite.

Export each comment into a table with these fields:

Field Purpose
comment_id Trace the original review item
file and line Locate the claimed problem
seed_id Match a known defect or mark none
claim Normalize the reviewer statement
reproduction Independent proof or disproof
severity Human-assigned impact
actionable Whether a concrete safe next step exists

Do not apply suggested changes. Reproduce each claim on the exact commit and run a focused deterministic check.

Step 5: run the Balanced review

Keep the PR diff, base branch, instructions, tools and settings unchanged. Request a new Copilot review and select Balanced. Confirm that the label on the resulting overview and timeline is Balanced rather than assuming the requested value took effect.

Repeat the same normalization and verification process. A differently worded comment about the same root cause counts as one detection, not an additional finding. If a comment combines two independently testable defects, split it into two evaluation rows while retaining the source comment ID.

Step 6: score detection quality

Calculate metrics separately for each effort level:

  • Seed recall: verified seeded defects found divided by total eligible seeds.
  • Precision: verified true findings divided by all actionable findings.
  • Critical recall: verified high-impact seeds found divided by eligible high-impact seeds.
  • Actionability: findings with a usable reproduction or verification step.
  • False-positive rate: disproved findings divided by all findings.
  • Time and cost: completion duration, AI credits and Actions minutes.

A Balanced review that finds one additional cross-service defect may be worth the higher cost for release-critical changes. A routine documentation or isolated test-data update may not benefit. Use several representative PRs before setting policy; one trial is not a benchmark.

Step 7: test repeatability and re-review behavior

Run each level on three fresh copies of the same pull request or reset to identical commits. Compare root-cause detection, not wording. Record variance and confidence intervals rather than expecting deterministic prose.

The GitHub how-to notes that a re-review may repeat comments even when previous comments were resolved or downvoted. Test this deliberately after fixing one seed. Classify repeated, stale, newly valid and newly introduced comments. If automatic review is not configured for new pushes, request the re-review manually and record that workflow choice.

Step 8: test agentic-context fallback

Copilot code review uses GitHub Actions for agentic capabilities such as full-project context gathering. GitHub documents that a review can still be generated when those workflows fail, but it will lack the added agentic features. If GitHub-hosted runners are disabled, the review can fall back to a more limited mode unless self-hosted runners are configured.

In the disposable repository, perform one authorized trial with the agentic workflow unavailable. Verify that a review still appears, note the reduced context, and do not compare its result directly with a full-context run. Your evidence schema should include context_mode so a limited review cannot masquerade as a normal result.

Step 9: validate defaults and overrides

GitHub lets organization owners set a default effort for automatic reviews, and repository administrators can override that default. Use the automatic-review configuration guide to build this matrix:

Organization default Repository setting Expected automatic level
Lite Inherit Lite
Balanced Inherit Balanced
Balanced Lite override Lite
Lite Balanced override Balanced

For every row, verify the observed label in the pull request. Also test one manual selection and confirm that it applies only to that review rather than silently changing defaults.

Step 10: design a risk-based policy

Do not force Balanced on every repository because it sounds safer. Define objective routing rules. A practical starting policy is:

  • Lite for routine, isolated and low-risk changes with strong deterministic CI.
  • Balanced for authentication, authorization, payment, privacy, concurrency, migration, cross-service and release-infrastructure changes.
  • Balanced for test-framework changes that can hide failures across many suites.
  • Human specialist review for excluded files, regulated controls, architectural decisions and any unresolved high-impact finding.

Review AI-credit and Actions budgets before enabling automatic Balanced review broadly. Budget exhaustion can block reviews, so a merge policy must define what happens when Copilot review is unavailable. The fallback should be a named human process, not silent approval.

Step 11: keep fixes behind verification

The GitHub code review how-to describes applying suggestions or asking the cloud agent to fix feedback. Treat that as a new change requiring its own review. Inspect the diff, run the focused reproduction, execute the regression suite, check for unrelated edits, and request a fresh review if policy requires it.

A correct comment can lead to an incorrect patch. Never use the effort label as evidence that an auto-applied fix is safe.

QA rollout checklist

  • Eligible seeded defects and clean controls are documented privately
  • PR commit, base branch and review context are frozen
  • Excluded file types are tested and recorded separately
  • Lite and Balanced labels are verified in overview and timeline evidence
  • Every comment is reproduced or disproved independently
  • Duplicate wording is normalized to root causes
  • Recall, precision, actionability, variance, time and cost are measured
  • Limited-context fallback runs are clearly labeled
  • Organization inheritance and repository overrides are verified
  • Re-review repetition and stale comments are tested
  • Budget exhaustion has a human fallback
  • Suggested fixes receive diff review and deterministic regression testing
  • Human reviewers retain merge and release authority

Official sources

Effort levels become useful when QA turns them into a controlled experiment. Seed representative risks, hold context constant, verify the actual label, reproduce every comment, track cost and variance, and route review depth by change criticality. Balanced can provide deeper analysis, but only evidence and human judgment can decide whether a pull request is ready.