70 pre-built scenarios

Test Case Library

Browse, filter, and select from curated test cases for evaluating AI systems across all dimensions.

Sign in

Access your evaluation projects.

or

Forgot your password?

Don't have an account? Create one

Create account

Start evaluating your AI systems.

or skip the password

Already have an account? Sign in

Reset password

Enter your email and we'll send you a reset link.

Back to sign in

Choose a new password

Enter and confirm your new password below.

Evaluation Builder

Define your system, select criteria, choose test cases, and run your evaluation.

1
System
2
Criteria
3
Test Cases
4
Evaluate
5
Results

Define Your System

Select Evaluation Criteria

These five dimensions define what "good" looks like for your AI system. We've pre-set the weights based on your system type - higher weight means that dimension counts more toward your overall score.

Hover the ⓘ icon next to each criterion to learn what it measures and when you might want to adjust its weight.

Select Test Cases

We've suggested test cases based on your system type. Select the ones you want to include, or add your own.

Run Evaluation

For each test case, paste the actual system output and score the result.

0 / 0 scored

Your Projects

+ New Evaluation
EU AI Act

EU AI Act Compliance Audit

Pre-configured evaluation template with all high-risk test cases mapped to EU AI Act obligations. Produces structured test evidence to support the testing and accuracy parts of your Articles 9-15 technical documentation.

Includes 54+ high-risk test cases across all 5 evaluation dimensions

Back to Dashboard

Compare Evaluations

Session Activity

Your wizard activity in this tab. This log stays in your browser and clears when you refresh or close the page. See our Privacy Policy for the site analytics we do collect.

Blog

Insights on AI evaluation

Practical guides, frameworks, and best practices for testing AI systems before production.

Views expressed in these articles reflect the EvalSpec team's experience and research. They do not constitute professional, legal, or regulatory advice.

← Back to all posts

Your AI Passed the Benchmark. Does It Follow Your Policies?

Most AI teams run evaluations. Fewer teams evaluate what actually matters: whether their system behaves the way they intended. A new open-source tool from Microsoft is starting to close that gap.

On 2 June 2026, Microsoft released ASSERT, short for Adaptive Spec-driven Scoring for Evaluation and Regression Testing. It converts plain-language descriptions of intended behaviours, policies, and constraints into structured test cases. Those cases are then run against the system, scored, and logged so teams can trace exactly where failures occur.

Sarah Bird, Chief Product Officer of Responsible AI at Microsoft, put it plainly: "If you don't understand the behaviour of the AI system, it's really hard to know if it's meeting your organisation's bar."

Why General Benchmarks Are Not Enough

The release lands at a pointed moment for the AI evaluation field.

Earlier this year, UC Berkeley researchers demonstrated that eight major AI agent benchmarks, including SWE-bench Verified and WebArena, could be exploited to return near-perfect scores without solving a single task. On some benchmarks, a 10-line script was enough to rewrite every test result to "passed." The exploit worked not by gaming the model but by gaming the evaluation pipeline itself.

That finding exposed something the industry has been slow to accept. Standard benchmarks measure general capability. They do not measure whether a deployed system respects your organisation's specific policies, safety constraints, or domain requirements. Scoring well on a leaderboard tells you nothing about whether the system will behave correctly in your product.

Spec-Driven Evaluation in Practice

ASSERT takes a different approach. Instead of testing against a fixed benchmark, teams write natural-language descriptions of how their system should behave. The framework then generates test scenarios from those descriptions, executes them, and produces scored results with intermediate logs.

This matters at three stages: pre-deployment testing before a system goes live, regression testing when models or prompts change, and continuous monitoring in production. Teams can encode policies directly into the evaluation layer rather than relying on a third party's benchmark to capture their requirements.

The framework is open source and available on GitHub. It is designed for developers building AI agents or any system where application-specific behaviour needs to be validated rather than general performance measured.

What This Means for QA Teams

If your current AI evaluation process consists of running off-the-shelf benchmarks and checking aggregate scores, you are probably missing the most important question: does this system actually do what we built it to do?

Spec-driven evaluation is not new as a concept. But tooling that makes it practical for product teams has been thin. ASSERT is a meaningful addition to that space.

EvalSpec helps teams build structured evaluation programmes that go beyond benchmark scores. If you are thinking about how to evaluate AI behaviour against your own requirements, start with EvalSpec.

Try EvalSpec free - evaluate your AI system across all 5 dimensions. No code required.

Start your free evaluation

Testing an AI system yourself?

Get the 70 test cases behind this framework, indexed by dimension and difficulty, or run a free evaluation with your own AI system.

The information in this article is provided for general guidance only. EvalSpec is a diagnostic evaluation tool and does not provide legal advice, regulatory certification, or compliance guarantees. For legal or regulatory questions, consult a qualified professional.

404

Page not found

The page you're looking for doesn't exist.

Back to home