70 pre-built scenarios

Test Case Library

Browse, filter, and select from curated test cases for evaluating AI systems across all dimensions.

Sign in

Access your evaluation projects.

or

Forgot your password?

Don't have an account? Create one

Create account

Start evaluating your AI systems.

or skip the password

Already have an account? Sign in

Reset password

Enter your email and we'll send you a reset link.

Back to sign in

Choose a new password

Enter and confirm your new password below.

Evaluation Builder

Define your system, select criteria, choose test cases, and run your evaluation.

1
System
2
Criteria
3
Test Cases
4
Evaluate
5
Results

Define Your System

Select Evaluation Criteria

These five dimensions define what "good" looks like for your AI system. We've pre-set the weights based on your system type - higher weight means that dimension counts more toward your overall score.

Hover the ⓘ icon next to each criterion to learn what it measures and when you might want to adjust its weight.

Select Test Cases

We've suggested test cases based on your system type. Select the ones you want to include, or add your own.

Run Evaluation

For each test case, paste the actual system output and score the result.

0 / 0 scored

Your Projects

+ New Evaluation
EU AI Act

EU AI Act Compliance Audit

Pre-configured evaluation template with all high-risk test cases mapped to EU AI Act obligations. Produces structured test evidence to support the testing and accuracy parts of your Articles 9-15 technical documentation.

Includes 54+ high-risk test cases across all 5 evaluation dimensions

Back to Dashboard

Compare Evaluations

Session Activity

Your wizard activity in this tab. This log stays in your browser and clears when you refresh or close the page. See our Privacy Policy for the site analytics we do collect.

Blog

Insights on AI evaluation

Practical guides, frameworks, and best practices for testing AI systems before production.

Views expressed in these articles reflect the EvalSpec team's experience and research. They do not constitute professional, legal, or regulatory advice.

← Back to all posts

The Week Frontier Models Met a Fairer Test, and Dropped by Thirty Points

On 22 September, Anthropic released Claude Opus 5.5. OpenAI answered the same afternoon with GPT-6 Sol and GPT-6 Luna. Nine models had already pushed Terminal-Bench 2.1, the standard test of how well an AI agent handles real command-line work, past 87 to 92 percent. The benchmark's maintainers retired it days later. Its replacement, Terminal-Bench 4.0, is harder and fairer. On it, the same frontier models top out at 59.6 percent. That is not a story about weaker models. It is a story about how much a passing score was hiding.

A leaderboard number tells you less than you think

Terminal-Bench 4.0 was built by the Laude Institute with Stanford researchers and open source contributors. It covers 66 tasks across software, machine learning, science, operations, security, hardware and media work, and it deliberately removed eight tasks from the old version that were saturated, refusal-prone, publicly solved or affected by unresolved quality issues. Claude Opus 5.5 and GPT-6 Astra now share the top score, at 59.6 percent.

That reset matters alongside research from UC Berkeley's Centre for Responsible, Decentralized Intelligence, published in April this year. Researchers built a scanning agent that audited eight major AI agent benchmarks, including SWE-bench, Terminal-Bench, WebArena and GAIA. Every single one could be exploited to near-perfect scores without solving a single task, through weaknesses such as shipped answer keys, weak string matching and judges that trust unverified output. A benchmark score tells you how a model plays one specific, static test. It tells you far less about whether it will do the job reliably in your own environment.

Passing once is not the same as working

Separate research from Microsoft's Copilot Studio team, published in August, makes the reliability gap concrete. Testing GPT-5.4 on 507 stateful business tasks, agents succeeded on the first attempt 65.36 percent of the time, but succeeded on all 20 repeated attempts only 25.25 percent of the time. Most of that failure, 77.5 percent, happened at the tool-call layer rather than in the final answer. An agent that can find a working path once is not the same as an agent that takes that path consistently. For anyone deploying agents in production, a single pass@1 figure on a vendor's leaderboard tells you almost nothing about whether the agent will hold up on Tuesday, then Wednesday, then a hundred times after that.

Governance is starting to catch up

On 1 September, OWASP's GenAI Security Project refreshed its 2026 Top 10 for LLM Applications and moved Excessive Agency up to the third-ranked threat. Alongside it, the project accepted a new Agent Control Standard aimed at runtime enforcement, not just pre-deployment testing. That direction matches what the benchmark data shows. Evaluation cannot be a one-off gate before launch. It has to run continuously, and it has to constrain what an agent is allowed to do, not only check what it did afterwards.

If the only evidence you have that your agent works is a single leaderboard run, you have less evidence than you think. EvalSpec does not run or trace agents today. It scores the responses your system gives to a fixed set of test cases, and stores each result with a record hash so it cannot be quietly changed afterwards.

Try EvalSpec free - evaluate your AI system across all 5 dimensions. No code required.

Start your free evaluation

Testing an AI system yourself?

Get the 70 test cases behind this framework, indexed by dimension and difficulty, or run a free evaluation with your own AI system.

The information in this article is provided for general guidance only. EvalSpec is a diagnostic evaluation tool and does not provide legal advice, regulatory certification, or compliance guarantees. For legal or regulatory questions, consult a qualified professional.

404

Page not found

The page you're looking for doesn't exist.

Back to home