OpenAI published results on August 27, 2026 from a randomized experiment showing that ChatGPT access and critical-thinking training improved work in different, complementary ways. ChatGPT helped participants produce more polished, coherent and expert-like answers, while causal-reasoning training led to a wider range of more distinct ideas.
The research studied more than 1,000 first-year Bocconi University students working on a real-world business case. They were assigned to one of four groups: ChatGPT access using GPT-4o, causal-reasoning training, both, or neither. Human graders scored submissions with a five-point rubric, while automated analysis measured idea count, variety, causal reasoning and similarity to expert work.
What the experiment found
- ChatGPT improved conventional quality: participants with access scored almost one point higher on the five-point rubric and produced clearer, more expert-like recommendations.
- Reasoning training broadened ideas: participants generated more distinct approaches and explained more clearly why an idea might work or fail.
- The standard rubric missed part of the gain: greater originality from reasoning training did not raise rubric scores because the rubric emphasized two conventional marketing goals.
- The combination was strongest across measures: students receiving both interventions retained the quality gains while also showing broader ideas and stronger questioning of assumptions.
These findings come from one educational task and should not be generalized into a universal model benchmark. They do, however, demonstrate how an evaluation can reward polish while overlooking diversity of thought.
Why this matters for QA engineers
AI QA evaluation rubrics often favor correctness, readability and completion. Those signals matter, but they can make an AI-generated test plan look strong even when it repeats familiar happy paths, misses alternative hypotheses or fails to challenge the requirement.
- Score coverage diversity: track distinct risk classes, state transitions, user roles, data boundaries and failure modes—not just the number of generated tests.
- Reward assumption checks: require the assistant or agent to state what must be true, what evidence would disprove it and where the requirement is ambiguous.
- Separate polish from substance: grade formatting and clarity independently from defect yield, mutation score, requirement coverage and uniqueness.
- Keep expert review in the loop: use human exploratory testing to add adversarial and domain-specific angles that a conventional rubric may not surface.
- Test the combined workflow: compare AI-only, human-only and human-plus-AI test design on the same seeded defects and unseen requirements.
A practical evaluation update
Add two explicit dimensions to the next AI test-generation benchmark: novel risk coverage and reasoned challenge. For each generated suite, ask reviewers to identify genuinely distinct failure hypotheses and to score whether the rationale explains when each hypothesis would or would not hold.
Then validate the output against seeded defects or mutation testing. This prevents fluent wording from becoming a proxy for effectiveness and gives teams evidence about whether AI expands coverage or merely improves presentation.
Bottom line
OpenAI’s study does not test software QA directly, but its evaluation lesson transfers cleanly: polished output and broad reasoning are different qualities. QA teams adopting ChatGPT or coding agents should measure both, then verify results against real defects instead of trusting a tidy final answer.