Visual Studio Code 1.146 adds an experimental option that lets an agent discover available models, launch comparisons, and read the resulting evidence to summarize findings. The feature is off by default and requires both Compare Models and the agent-comparison tools setting to be enabled.

This is useful for QA teams evaluating AI coding assistants on their own representative work—not as a shortcut to accepting a model ranking. Microsoft’s documentation is explicit: a judge recommendation is not proof of correctness, and code still needs review and project tests before integration.

What changed in VS Code 1.146

In the Agents window, Compare Models can run the same bounded task across multiple attempts, optionally use a judge, and support later synthesis. The new experimental agent tools can discover models, start comparisons, and read results. For repository work, attempts run in separate Git worktrees from committed state; uncommitted changes in the original working tree are not included.

That isolation is valuable, but it has a material testing boundary: comparison worktree sessions use Allow all, which skips tool-approval prompts. Worktrees isolate edits; they do not by themselves limit machine or network access.

Why this matters for QA engineers

Parallel model runs can turn a vague preference into a testable evaluation. They can also multiply variability, model usage, and permissions. Treat each comparison as a controlled experiment: same baseline, same acceptance criteria, independent verification, and a clear record of what was actually measured.

  • Freeze the baseline. Start from a committed fixture, record the commit SHA, and keep the prompt, repository instructions, tools, harness, permissions, and test command identical across attempts.
  • Use bounded tasks. Prefer a small defect or feature with observable acceptance criteria over an open-ended refactor. Capture changed files, test output, lint results, and any failures per attempt.
  • Test the safety boundary. Before enabling an agent-driven comparison, review its effective permissions and sandboxing. Do not equate separate worktrees with restricted external access.
  • Challenge the judge. Include seeded edge cases or a hidden regression test. Compare its recommendation against clean-checkout CI and human review; the judge scorecard is evidence to inspect, not an approval gate.
  • Keep synthesis separate. A synthesized result is a new implementation in another worktree and does not automatically merge, push, or create a pull request. Run the same review and validation again.
  • Measure the whole trade-off. Track pass rate, defect escapes, elapsed time, and usage. The documentation notes that token values can be lower bounds and the scorecard does not measure credit or currency cost.

A practical first pilot

Choose one stable repository and two or three approved models. Give every attempt one small task and the same test command. Preserve the resulting worktrees long enough to review the diffs and logs, then compare clean-CI outcomes rather than chat quality alone. If the winner cannot reproduce its result from a clean checkout, it has not won the QA evaluation.

Sources