Most AI teams run evaluations. Fewer teams evaluate what actually matters: whether their system behaves the way they intended. A new open-source tool from Microsoft is starting to close that gap.
On 2 June 2026, Microsoft released ASSERT, short for Adaptive Spec-driven Scoring for Evaluation and Regression Testing. It converts plain-language descriptions of intended behaviours, policies, and constraints into structured test cases. Those cases are then run against the system, scored, and logged so teams can trace exactly where failures occur.
Sarah Bird, Chief Product Officer of Responsible AI at Microsoft, put it plainly: "If you don't understand the behaviour of the AI system, it's really hard to know if it's meeting your organisation's bar."
Why General Benchmarks Are Not Enough
The release lands at a pointed moment for the AI evaluation field.
Earlier this year, UC Berkeley researchers demonstrated that eight major AI agent benchmarks, including SWE-bench Verified and WebArena, could be exploited to return near-perfect scores without solving a single task. On some benchmarks, a 10-line script was enough to rewrite every test result to "passed." The exploit worked not by gaming the model but by gaming the evaluation pipeline itself.
That finding exposed something the industry has been slow to accept. Standard benchmarks measure general capability. They do not measure whether a deployed system respects your organisation's specific policies, safety constraints, or domain requirements. Scoring well on a leaderboard tells you nothing about whether the system will behave correctly in your product.
Spec-Driven Evaluation in Practice
ASSERT takes a different approach. Instead of testing against a fixed benchmark, teams write natural-language descriptions of how their system should behave. The framework then generates test scenarios from those descriptions, executes them, and produces scored results with intermediate logs.
This matters at three stages: pre-deployment testing before a system goes live, regression testing when models or prompts change, and continuous monitoring in production. Teams can encode policies directly into the evaluation layer rather than relying on a third party's benchmark to capture their requirements.
The framework is open source and available on GitHub. It is designed for developers building AI agents or any system where application-specific behaviour needs to be validated rather than general performance measured.
What This Means for QA Teams
If your current AI evaluation process consists of running off-the-shelf benchmarks and checking aggregate scores, you are probably missing the most important question: does this system actually do what we built it to do?
Spec-driven evaluation is not new as a concept. But tooling that makes it practical for product teams has been thin. ASSERT is a meaningful addition to that space.
EvalSpec helps teams build structured evaluation programmes that go beyond benchmark scores. If you are thinking about how to evaluate AI behaviour against your own requirements, start with EvalSpec.
Try EvalSpec free - evaluate your AI system across all 5 dimensions. No code required.
Start your free evaluation