GitHub announced on November 17, 2026 that GitHub Copilot Evals is available in public preview. The feature is designed to help users create repeatable tests for the skills, custom agents, and MCP tools in their Copilot setup, then compare runs after a customization changes.
For QA teams, GitHub Copilot Evals QA is useful because a single convincing agent run is not a regression strategy. An agent can vary its approach and output from run to run. A practical evaluation needs a bounded task, observable checks, and preserved evidence of both the result and the route the agent took.
What GitHub announced
GitHub describes an eval as a repeatable test for AI behavior: provide a realistic task, define what a good result looks like, and check the result and the process. Users can create an eval for a skill, custom agent, or MCP tool; run it after approving the run; inspect passed and failed checks; and compare results before and after a customization. GitHub says Copilot Evals uses the extensible local tool Vally as its evaluation engine.
The preview is available for Copilot Individual, Business, and Enterprise. GitHub documents installation paths for Visual Studio Code, Copilot CLI, and the Copilot app. It is a preview feature, so teams should validate its behavior in the exact client and project configuration they plan to use.
Why this matters for QA engineers
Prompts, models, tools, and agent instructions are all moving parts. A change that improves a happy-path response can quietly weaken a permission boundary, skip a test, or change an integration call. Repeatable evals give those changes a test surface, but the checks must measure more than fluent prose.
- Freeze a small, realistic fixture. Start with a reproducible repository, issue, log, or synthetic service record. Pin the revision and include a clear success condition so a passing run is attributable to the evaluated configuration.
- Split outcome checks from process checks. Verify the final diff, test output, or remote state separately from evidence such as tools invoked, files changed, permissions requested, and whether the agent stayed within scope.
- Include negative cases. Seed an unavailable dependency, ambiguous requirement, revoked credential, or prohibited write. The expected result can be a safe stop or clarification; an unsupported workaround should fail.
- Compare against a baseline. Run the same eval before and after a prompt, skill, model, or MCP change. Record the exact configuration and fixture revision alongside each result.
- Challenge the evaluator. Use near-miss outputs, superficial passing tests, and plausible but incorrect explanations. A check that accepts these cases is not a dependable release gate.
- Keep independent evidence. Re-run tests in a clean checkout and query any affected external system directly. The agent transcript is useful context, not proof that a change is correct.
A compact first eval
Choose one agent behavior that matters to your team. For example, give a custom debugging agent a synthetic failing test with a unique error marker and require it to identify the failing component, propose a minimal patch, run the targeted test, and avoid changing an unrelated protected file. Define checks for the marker, the allowed file list, a passing test command, and a clean independent test run.
Then add a negative variant: remove the dependency needed to run the test. The acceptance criterion should be an explicit blocker report and no fabricated success claim, not a patch. This kind of paired case helps detect regressions where an agent becomes more willing to act but less reliable about its limits.
Treat evals as release evidence, not a scorecard
Keep the first suite small and version it with the customization it tests. Track failures by category—incorrect outcome, missing evidence, unauthorized action, unsafe refusal, or flaky environment—rather than reducing every run to one aggregate score. Expand the suite only when each new case maps to a genuine workflow risk.
GitHub’s announcement explains how to create, run, inspect, and compare evals; it does not guarantee a given agent’s correctness or the sufficiency of any particular check. QA teams should still review code, enforce environment controls, and validate final results independently.
Source
- GitHub Changelog: GitHub Copilot Evals is now in public preview (November 17, 2026; availability, workflow, approval requirement, installation paths, and Vally evaluation engine)
