70 pre-built scenarios

Test Case Library

Browse, filter, and select from curated test cases for evaluating AI systems across all dimensions.

Sign in

Access your evaluation projects.

or

Forgot your password?

Don't have an account? Create one

Create account

Start evaluating your AI systems.

or skip the password

Already have an account? Sign in

Reset password

Enter your email and we'll send you a reset link.

Back to sign in

Choose a new password

Enter and confirm your new password below.

Evaluation Builder

Define your system, select criteria, choose test cases, and run your evaluation.

1
System
2
Criteria
3
Test Cases
4
Evaluate
5
Results

Define Your System

Select Evaluation Criteria

These five dimensions define what "good" looks like for your AI system. We've pre-set the weights based on your system type - higher weight means that dimension counts more toward your overall score.

Hover the ⓘ icon next to each criterion to learn what it measures and when you might want to adjust its weight.

Select Test Cases

We've suggested test cases based on your system type. Select the ones you want to include, or add your own.

Run Evaluation

For each test case, paste the actual system output and score the result.

0 / 0 scored

Your Projects

+ New Evaluation
EU AI Act

EU AI Act Compliance Audit

Pre-configured evaluation template with all high-risk test cases mapped to EU AI Act obligations. Produces structured test evidence to support the testing and accuracy parts of your Articles 9-15 technical documentation.

Includes 54+ high-risk test cases across all 5 evaluation dimensions

Back to Dashboard

Compare Evaluations

Session Activity

Your wizard activity in this tab. This log stays in your browser and clears when you refresh or close the page. See our Privacy Policy for the site analytics we do collect.

Blog

Insights on AI evaluation

Practical guides, frameworks, and best practices for testing AI systems before production.

Views expressed in these articles reflect the EvalSpec team's experience and research. They do not constitute professional, legal, or regulatory advice.

← Back to all posts

Your LLM Passed Every Eval. Your Agent Didn't.

AI agents are shipping fast. QA is catching up slowly. Most teams are still running the same LLM evaluation suites they built for chatbots, then attaching them to systems that can write to databases, trigger deployments, and call external APIs. The gap between what those evals catch and what agents can actually do in production is where the real risk lives.

OWASP has named that gap. In 2026, the Open Worldwide Application Security Project published its first Top 10 for Agentic Applications, a peer-reviewed framework identifying the most critical security risks facing autonomous AI systems. It covers territory that no standard LLM benchmark touches.

Ten risks your current eval suite probably misses

The OWASP Top 10 for Agentic Applications introduces a new vocabulary for agentic failure modes. A few stand out as immediately relevant for QA and product teams.

Agent Goal Hijack (ASI01) is the agentic version of prompt injection. An attacker embeds malicious instructions in a ticket, a wiki page, or a retrieved document. The agent reads it, treats it as instruction, and acts on it. Your eval probably tests adversarial prompts in isolation. It almost certainly does not test what happens when the adversarial content arrives through a tool call.

Memory and Context Poisoning (ASI06) targets the persistent memory stores that give agents continuity across sessions. Corrupted context shapes future behaviour in ways that are hard to detect and harder to trace. Static benchmarks cannot catch this because they reset state between runs.

Cascading Agent Failures (ASI08) describes how a small error in one agent propagates through connected systems. In a multi-agent architecture, a single compromised component can affect every downstream decision. Evals that test agents in isolation will never surface this class of failure.

Research published in May by Help Net Security adds another dimension: "confused-deputy" attacks, where an authorised system is manipulated into misusing its own privileges. The research found that prompt-only defences are brittle under adversarial conditions and advocates a strict "propose-commit split" architecture, where the model can reason and draft but cannot execute writes without a separate policy gate and human approval step.

What to test first

If your team is shipping agentic features, three test areas should move to the top of your backlog.

First, test goal persistence under indirect injection. Feed the agent adversarial instructions embedded in retrieved content, not in the direct prompt. Observe whether its objective shifts.

Second, stress-test memory across sessions. Introduce corrupted or contradictory entries into the agent's memory store and evaluate whether downstream reasoning degrades.

Third, map your tool permissions. Most agents have more access than they need. Applying least-privilege to tool grants is one of the highest-value controls available before any eval is written.

OWASP's framework is a starting point, not a complete test plan. But it is the most authoritative signal yet that agentic evaluation requires its own discipline, separate from LLM testing and separate from traditional software QA.

If you are building that test plan, EvalSpec is designed to help.

Try EvalSpec free - evaluate your AI system across all 5 dimensions. No code required.

Start your free evaluation

Testing an AI system yourself?

Get the 70 test cases behind this framework, indexed by dimension and difficulty, or run a free evaluation with your own AI system.

The information in this article is provided for general guidance only. EvalSpec is a diagnostic evaluation tool and does not provide legal advice, regulatory certification, or compliance guarantees. For legal or regulatory questions, consult a qualified professional.

404

Page not found

The page you're looking for doesn't exist.

Back to home