70 pre-built scenarios

Test Case Library

Browse, filter, and select from curated test cases for evaluating AI systems across all dimensions.

Sign in

Access your evaluation projects.

or

Forgot your password?

Don't have an account? Create one

Create account

Start evaluating your AI systems.

or skip the password

Already have an account? Sign in

Reset password

Enter your email and we'll send you a reset link.

Back to sign in

Choose a new password

Enter and confirm your new password below.

Evaluation Builder

Define your system, select criteria, choose test cases, and run your evaluation.

1
System
2
Criteria
3
Test Cases
4
Evaluate
5
Results

Define Your System

Select Evaluation Criteria

These five dimensions define what "good" looks like for your AI system. We've pre-set the weights based on your system type - higher weight means that dimension counts more toward your overall score.

Hover the ⓘ icon next to each criterion to learn what it measures and when you might want to adjust its weight.

Select Test Cases

We've suggested test cases based on your system type. Select the ones you want to include, or add your own.

Run Evaluation

For each test case, paste the actual system output and score the result.

0 / 0 scored

Your Projects

+ New Evaluation
EU AI Act

EU AI Act Compliance Audit

Pre-configured evaluation template with all high-risk test cases mapped to EU AI Act obligations. Produces structured test evidence to support the testing and accuracy parts of your Articles 9-15 technical documentation.

Includes 54+ high-risk test cases across all 5 evaluation dimensions

Back to Dashboard

Compare Evaluations

Session Activity

Your wizard activity in this tab. This log stays in your browser and clears when you refresh or close the page. See our Privacy Policy for the site analytics we do collect.

Blog

Insights on AI evaluation

Practical guides, frameworks, and best practices for testing AI systems before production.

Views expressed in these articles reflect the EvalSpec team's experience and research. They do not constitute professional, legal, or regulatory advice.

← Back to all posts

Your automated eval says the model failed. Did it?

Automated evaluation is fast. It is also, it turns out, rather excitable. A paper posted to arXiv on 10 October 2026 tested two frontier models against their own published safety claims. Automated classifiers flagged 312 possible failures. Human reviewers confirmed 19. That gap matters if you run AI QA.

What the researchers did

The authors built a method called LEST. It treats each safety claim in a model's system card as a specification, then tests that claim from the outside.

The trick is simple. Take a prompt the model refuses. Reword it, keeping the intent. See whether the model now complies.

They used 97 seed prompts and 1,836 mutated versions. Classifiers flagged 312 candidate failures. People confirmed 19, which is 6.1%.

Those 19 were still real. They contradicted 6 of the 9 claims where a refusal test was fair. So the failures exist. There are just far fewer than the automation suggested.

Two lessons for QA teams

  1. Put a human gate after the classifier. Treat automated flags as candidates, not verdicts. Over-reporting wastes triage time and teaches people to ignore the dashboard. Do not drop automation, though. It found the 19.
  2. Test against promises, not just benchmarks. Of 56 documented safety claims, 29 could be tested through prompts. Only 17 of those were exercised by a benchmark prompt. The rest had no coverage at all.

Write down what your system claims to do. Then map each test to a claim. Gaps show up quickly.

What does a passing test prove?

A separate survey from 4 October 2026 covers LLM-generated tests. Its authors make a neat point. A test that lifts coverage, a test that matches a reference program and a test that catches a requirements breach each prove something different.

They also note that execution feedback can help tests run while weakening the independence of the oracle. An oracle is whatever decides if the output is right. If it is not independent, a green tick means less than it looks.

A fair caveat

This is one study. It used two models and three risk categories. Treat the 312 to 19 ratio as a prompt to check your own pipeline, not a benchmark to quote.

Where next

If your evals flag more than your reviewers ever confirm, that is worth knowing. EvalSpec helps teams build evaluation specs that link every test to a claim and every flag to a human decision.

Try EvalSpec free - evaluate your AI system across all 5 dimensions in under an hour. No code required.

Start your free evaluation

Testing an AI system yourself?

Get the 70 test cases behind this framework, indexed by dimension and difficulty, or run a free evaluation with your own AI system.

The information in this article is provided for general guidance only. EvalSpec is a diagnostic evaluation tool and does not provide legal advice, regulatory certification, or compliance guarantees. For legal or regulatory questions, consult a qualified professional.

404

Page not found

The page you're looking for doesn't exist.

Back to home