70 pre-built scenarios

Test Case Library

Browse, filter, and select from curated test cases for evaluating AI systems across all dimensions.

Sign in

Access your evaluation projects.

or

Forgot your password?

Don't have an account? Create one

Create account

Start evaluating your AI systems.

or skip the password

Already have an account? Sign in

Reset password

Enter your email and we'll send you a reset link.

Back to sign in

Choose a new password

Enter and confirm your new password below.

Evaluation Builder

Define your system, select criteria, choose test cases, and run your evaluation.

1
System
2
Criteria
3
Test Cases
4
Evaluate
5
Results

Define Your System

Select Evaluation Criteria

These five dimensions define what "good" looks like for your AI system. We've pre-set the weights based on your system type - higher weight means that dimension counts more toward your overall score.

Hover the ⓘ icon next to each criterion to learn what it measures and when you might want to adjust its weight.

Select Test Cases

We've suggested test cases based on your system type. Select the ones you want to include, or add your own.

Run Evaluation

For each test case, paste the actual system output and score the result.

0 / 0 scored

Your Projects

+ New Evaluation
EU AI Act

EU AI Act Compliance Audit

Pre-configured evaluation template with all high-risk test cases mapped to EU AI Act obligations. Produces structured test evidence to support the testing and accuracy parts of your Articles 9-15 technical documentation.

Includes 54+ high-risk test cases across all 5 evaluation dimensions

Back to Dashboard

Compare Evaluations

Session Activity

Your wizard activity in this tab. This log stays in your browser and clears when you refresh or close the page. See our Privacy Policy for the site analytics we do collect.

Blog

Insights on AI evaluation

Practical guides, frameworks, and best practices for testing AI systems before production.

Views expressed in these articles reflect the EvalSpec team's experience and research. They do not constitute professional, legal, or regulatory advice.

← Back to all posts

Your Agent Got the Right Answer. That Doesn't Mean It Worked.

In March 2026, OpenAI announced it was acquiring Promptfoo, a small AI security and evaluation startup, and building its red-teaming tools straight into its agent platform. That's a significant marker. The best-funded AI lab in the world decided that testing agents for security and correctness isn't something to bolt on later; it's core infrastructure. Most QA teams are still testing agents the way they'd test a chatbot: check the final answer, move on. That gap is now the biggest blind spot in agentic AI.

The answer can be right and the process still broken

An AI agent doesn't just produce a reply. It plans steps, calls tools, retrieves information and reasons through several stages before it gives you anything to check. LangChain, the company behind widely used agent-building tools, argues testing should follow that whole path, not just the destination. It calls this trajectory evaluation: scoring the full sequence of decisions an agent makes across three areas. Whether it grounded its answer in the right information. Whether the interaction actually served the user's goal. Whether it stayed inside safety and security boundaries along the way.

Here's why that matters. An agent can reach a correct-looking answer by calling the wrong tool, retrieving the wrong document or taking a shortcut that happens to work this time and fails the next. Output-only testing can't see any of that. It only sees that the answer passed.

Test agents like distributed systems, not chatbots

A recent research paper from a twelve-author team makes a related point from a different angle. Most agent testing today is ad hoc: a handful of manually written prompts, checked by eye. The paper argues for structural testing instead, borrowing directly from decades of software engineering practice. That means building reproducible test harnesses, tracking an agent's state across multiple turns and validating every tool call rather than trusting that it worked.

The practical shift here is one of mindset. An agent behaves more like a distributed system than a script. It holds state, it calls services it doesn't fully control and it can fail in ways that only surface under specific sequences of events. Testing it like a distributed system, not a single function, is what catches those failures before a user does.

Security testing is no longer a bolt-on

The Promptfoo acquisition fits both points. OpenAI isn't just buying a red-teaming tool. It's embedding automated security and evaluation testing directly into its agent platform, so every agent built on it gets tested for vulnerabilities as standard, not as an optional extra a customer has to configure. For QA leads, that's a strong signal about where the baseline is heading. Security and correctness testing for agents won't stay a specialist add-on for much longer.

If your testing stops at whether the agent got the right answer, you're only checking the destination, not the journey. EvalSpec does not evaluate tool calls or traces today. It scores the final responses your system gives across five quality dimensions, which is one part of the picture this article describes.

Try EvalSpec free - evaluate your AI system across all 5 dimensions. No code required.

Start your free evaluation

Testing an AI system yourself?

Get the 70 test cases behind this framework, indexed by dimension and difficulty, or run a free evaluation with your own AI system.

The information in this article is provided for general guidance only. EvalSpec is a diagnostic evaluation tool and does not provide legal advice, regulatory certification, or compliance guarantees. For legal or regulatory questions, consult a qualified professional.

404

Page not found

The page you're looking for doesn't exist.

Back to home