70 pre-built scenarios

Test Case Library

Browse, filter, and select from curated test cases for evaluating AI systems across all dimensions.

Sign in

Access your evaluation projects.

or

Forgot your password?

Don't have an account? Create one

Create account

Start evaluating your AI systems.

or skip the password

Already have an account? Sign in

Reset password

Enter your email and we'll send you a reset link.

Back to sign in

Choose a new password

Enter and confirm your new password below.

Evaluation Builder

Define your system, select criteria, choose test cases, and run your evaluation.

1
System
2
Criteria
3
Test Cases
4
Evaluate
5
Results

Define Your System

Select Evaluation Criteria

These five dimensions define what "good" looks like for your AI system. We've pre-set the weights based on your system type - higher weight means that dimension counts more toward your overall score.

Hover the ⓘ icon next to each criterion to learn what it measures and when you might want to adjust its weight.

Select Test Cases

We've suggested test cases based on your system type. Select the ones you want to include, or add your own.

Run Evaluation

For each test case, paste the actual system output and score the result.

0 / 0 scored

Your Projects

+ New Evaluation
EU AI Act

EU AI Act Compliance Audit

Pre-configured evaluation template with all high-risk test cases mapped to EU AI Act obligations. Produces structured test evidence to support the testing and accuracy parts of your Articles 9-15 technical documentation.

Includes 54+ high-risk test cases across all 5 evaluation dimensions

Back to Dashboard

Compare Evaluations

Session Activity

Your wizard activity in this tab. This log stays in your browser and clears when you refresh or close the page. See our Privacy Policy for the site analytics we do collect.

Blog

Insights on AI evaluation

Practical guides, frameworks, and best practices for testing AI systems before production.

Views expressed in these articles reflect the EvalSpec team's experience and research. They do not constitute professional, legal, or regulatory advice.

← Back to all posts

Voice Agents Break Differently. Most Evaluation Frameworks Miss It.

Most teams evaluating voice agents are using the wrong toolkit. They have LLM evaluation frameworks, prompt testing pipelines, and accuracy benchmarks. What they don't have is a reliable way to measure whether the voice agent actually works in a real conversation.

That gap matters. Production data from over four million voice agent calls shows the industry median response time sits at 1.5 to 1.7 seconds. Human conversation expectation is 300 milliseconds. That is a fivefold shortfall, and standard LLM metrics won't surface it.

Why voice agents break in ways text agents don't

A voice agent is not a chatbot with a voice attached. It is a pipeline. Audio comes in, speech-to-text (STT) converts it, a language model processes the intent, and text-to-speech (TTS) delivers the response. Each stage introduces its own failure mode.

STT errors compound. A name transcribed incorrectly will cause the LLM to misroute the call, and the TTS output will confirm the error with perfect confidence. Standard accuracy metrics won't catch this: word error rate (WER) treats a missed filler word the same as a garbled customer name.

Barge-in handling is another dimension text frameworks ignore entirely. When a user interrupts mid-response, the agent must detect it within 200 milliseconds and change course. Miss that window and the experience breaks, even if every accuracy score looks healthy.

The metrics that actually matter

A structured evaluation covers four layers: infrastructure, agent execution, user behaviour, and business outcome. Most teams only track the middle two.

For latency, track percentiles rather than averages. Target P50 under 1.5 seconds and P95 under 3.5 seconds. An average can look acceptable while 10% of calls are taking five seconds or more.

For speech quality, WER should sit below 5% for enterprise production. Context matters more than raw numbers: a name error in a healthcare booking has far higher impact than a missed filler word.

First-call resolution (FCR) and containment rate are the business-level signals that tie technical performance to outcomes. A contact centre voice agent should resolve 75% of calls on first contact. Healthcare use cases demand hallucination rates below 0.5%.

Building a QA framework for voice

A structured pre-production testing split looks like this: 40% happy path scenarios, 30% edge cases, 15% error handling, 10% adversarial inputs, 5% acoustic variations. Automate 80% of evaluation. Reserve human review for edge cases and novel failures.

One finding from production data stands out as a warning: small prompt changes cause large quality swings in voice agents. Every prompt update should trigger a full test run before deployment. And every production failure should become a permanent regression test case, not a one-off fix.

The trade-off to keep in mind: a perfectly accurate agent that responds in five seconds produces worse business outcomes than a slightly less accurate agent responding in one second. Your evaluation framework needs to reflect that.

EvalSpec does not measure audio, latency or speech quality. It can score the text responses behind a voice agent across five quality dimensions, and compare runs of the same system over time.

Try EvalSpec free - evaluate your AI system across all 5 dimensions. No code required.

Start your free evaluation

Testing an AI system yourself?

Get the 70 test cases behind this framework, indexed by dimension and difficulty, or run a free evaluation with your own AI system.

The information in this article is provided for general guidance only. EvalSpec is a diagnostic evaluation tool and does not provide legal advice, regulatory certification, or compliance guarantees. For legal or regulatory questions, consult a qualified professional.

404

Page not found

The page you're looking for doesn't exist.

Back to home