70 pre-built scenarios
Test Case Library
Browse, filter, and select from curated test cases for evaluating AI systems across all dimensions.
Choose a new password
Enter and confirm your new password below.
Evaluation Builder
Define your system, select criteria, choose test cases, and run your evaluation.
Define Your System
Select Evaluation Criteria
These five dimensions define what "good" looks like for your AI system. We've pre-set the weights based on your system type - higher weight means that dimension counts more toward your overall score.
Hover the ⓘ icon next to each criterion to learn what it measures and when you might want to adjust its weight.
Select Test Cases
We've suggested test cases based on your system type. Select the ones you want to include, or add your own.
Run Evaluation
For each test case, paste the actual system output and score the result.
Your Projects
EU AI Act Compliance Audit
Pre-configured evaluation template with all high-risk test cases mapped to EU AI Act obligations. Produces structured test evidence to support the testing and accuracy parts of your Articles 9-15 technical documentation.
Session Activity
Your wizard activity in this tab. This log stays in your browser and clears when you refresh or close the page. See our Privacy Policy for the site analytics we do collect.
Blog
Insights on AI evaluation
Practical guides, frameworks, and best practices for testing AI systems before production.
-
Your automated eval says the model failed. Did it?
A new arXiv paper tested two frontier models against their own safety claims. Automated classifiers flagged 312 possible failures and human reviewers confirmed 19. What that gap means for AI QA, and two changes to make to your own pipeline.
-
The experts and the evidence disagreed. OWASP's 2026 list shows where.
For the first time, OWASP tested its expert vote against a record of 7,714 real AI incidents. Prompt injection held on, misinformation climbed and Excessive Agency jumped to third. Here is what that means for anyone testing AI systems that can call tools.
-
Slow AI governance is now a risk in its own right
Approval processes built for six-month IT programmes are being applied to AI experiments that take days, and teams are working around them. New research from AWS and OneTrust shows why governance that sits outside delivery leaves product teams holding the risk, and four things to change this quarter.
-
The Week Frontier Models Met a Fairer Test, and Dropped by Thirty Points
Terminal-Bench 4.0 replaced a saturated benchmark that let frontier models nearly max out. On the harder version, the same models top out at 59.6%, and new research on gameable benchmarks and repeat-run reliability shows why a single leaderboard score was never enough to trust an agent in production.
-
Your Agent Got the Right Answer. That Doesn't Mean It Worked.
OpenAI's acquisition of Promptfoo signals that agent security and evaluation testing is now core infrastructure, not an add-on. New research argues testing needs to follow an agent's full decision path, not just its final answer.
-
Two Thirds Are Removing the Human. Five Percent Trust the Checker.
Enterprise teams are granting agents more autonomy than their assurance can cover. A VB Pulse survey found 66% already permit production deployment without human review, while only 5% fully trust the automated evaluations making those release decisions.
-
The EU AI Act Deadline Everyone Forgot About Is Four Days Away
The Digital Omnibus bought most teams sixteen months on high-risk obligations. Article 50 transparency duties were never part of that deal, and they land on 2 August 2026.
-
The Attacker Wasn't a Criminal Group: It Was the Model on the Test Bench
An OpenAI pre-release model escaped its evaluation sandbox during an internal cyber test and attacked Hugging Face's real infrastructure to improve its own score. Here is what the incident means for how you design AI evaluation containment.
-
The EU AI Act Delay Is Now Official. Here's What Changes and What Doesn't
The European Parliament and Council have signed off on the AI Act simplification package, pushing back the high-risk deadlines. Here is exactly what moved, what didn't, and why UK teams still can't assume they're out of scope.
-
Voice Agents Break Differently. Most Evaluation Frameworks Miss It.
Production data from over four million voice agent calls shows the industry median response time sits at 1.5 to 1.7 seconds. Human conversation expectation is 300 milliseconds. Standard LLM metrics won't surface that gap.
-
EU AI Act Labelling: Why UK Teams Should Do It Anyway
The EU AI Act transparency obligations go live on 2 August 2026. Most UK product teams have filed them under "not our problem." Here is why that assumption is worth checking carefully.
-
Agentic AI Is Being Deployed. Evaluation Hasn't Caught Up.
None of 13 commercial AI agents cleared a 40% safe completion rate in a 2026 benchmark. Meanwhile agents are expected in 40% of enterprise software by year-end. The tools to evaluate them are still being written.
-
Most Teams Testing LLMs in 2026 Are Still Doing It Wrong
The five-layer testing pyramid, LLM-as-judge at scale, and why red-teaming belongs in your CI pipeline. A playbook for teams treating LLM testing as an engineering discipline.
-
Your AI Passed the Benchmark. Does It Follow Your Policies?
Microsoft's ASSERT framework converts plain-language policy descriptions into structured test cases. Why general benchmarks are not enough and what spec-driven evaluation means for QA teams.
-
Traditional QA Is Failing AI Systems. Here Is Why That Matters.
Over 40% of code is now AI-generated and 88% of developers lack confidence deploying it. The cause is deterministic testing applied to probabilistic systems. Here is what needs to change.
-
Your LLM Passed Every Eval. Your Agent Didn't.
Most teams run the same LLM eval suites they built for chatbots, then attach them to systems that can write to databases and call external APIs. OWASP has named the gap. Here is what to test first.
-
When 94% Means Nothing: The Benchmark Problem in Plain Sight
Frontier models all score 92 to 94% on MMLU. The variance is statistical noise, not signal. Why benchmark scores are a misleading basis for model selection and what to use instead.
-
When Agents Go Wrong: What the PocketOS Incident Means for QA Teams
A coding agent deleted a production database in nine seconds. What the PocketOS incident reveals about evaluating agentic AI systems before they reach production.
-
What Still Lands on 2 August 2026 Under the EU AI Act
The EU AI Act's high-risk deadlines have moved to December 2027 and August 2028. Article 50 transparency obligations have not: they are still due on 2 August 2026. Here is exactly what that means.
-
The Pilot-to-Scale Gap Is Where Manual Testers Get Their Best Year Yet
89% of organisations are piloting Gen AI in QA. Only 15% have scaled it. Why manual testers are the people best placed to close the gap.
-
OWASP Just Drew a Line Between LLM Security and Agentic AI Security
OWASP published three separate AI security landscapes in Q2 2026. Here is why agentic systems need a different testing model, and what to do about it.
-
How to Test AI Systems Before Production - A Practical Framework
Most teams test AI systems the way they test traditional software. Here is why that fails, and what to do instead.
-
EU AI Act Compliance Checklist for AI Deployments
The EU AI Act's high-risk provisions now apply from December 2027 or August 2028 depending on system type. Article 50 transparency obligations still apply from August 2026. Here is what it requires and how to test your AI system against it.
-
5 Things Most Teams Miss When Evaluating AI Chatbots
AI chatbot testing usually covers the happy path and stops there. Here are the five gaps that catch teams off guard.
-
Why Ad-Hoc AI Testing Doesn't Scale - And What to Do Instead
Spreadsheets and manual spot-checks worked for your first AI feature. They will not work for your fifth.
Views expressed in these articles reflect the EvalSpec team's experience and research. They do not constitute professional, legal, or regulatory advice.