Most teams evaluating voice agents are using the wrong toolkit. They have LLM evaluation frameworks, prompt testing pipelines, and accuracy benchmarks. What they don't have is a reliable way to measure whether the voice agent actually works in a real conversation.
That gap matters. Production data from over four million voice agent calls shows the industry median response time sits at 1.5 to 1.7 seconds. Human conversation expectation is 300 milliseconds. That is a fivefold shortfall, and standard LLM metrics won't surface it.
Why voice agents break in ways text agents don't
A voice agent is not a chatbot with a voice attached. It is a pipeline. Audio comes in, speech-to-text (STT) converts it, a language model processes the intent, and text-to-speech (TTS) delivers the response. Each stage introduces its own failure mode.
STT errors compound. A name transcribed incorrectly will cause the LLM to misroute the call, and the TTS output will confirm the error with perfect confidence. Standard accuracy metrics won't catch this: word error rate (WER) treats a missed filler word the same as a garbled customer name.
Barge-in handling is another dimension text frameworks ignore entirely. When a user interrupts mid-response, the agent must detect it within 200 milliseconds and change course. Miss that window and the experience breaks, even if every accuracy score looks healthy.
The metrics that actually matter
A structured evaluation covers four layers: infrastructure, agent execution, user behaviour, and business outcome. Most teams only track the middle two.
For latency, track percentiles rather than averages. Target P50 under 1.5 seconds and P95 under 3.5 seconds. An average can look acceptable while 10% of calls are taking five seconds or more.
For speech quality, WER should sit below 5% for enterprise production. Context matters more than raw numbers: a name error in a healthcare booking has far higher impact than a missed filler word.
First-call resolution (FCR) and containment rate are the business-level signals that tie technical performance to outcomes. A contact centre voice agent should resolve 75% of calls on first contact. Healthcare use cases demand hallucination rates below 0.5%.
Building a QA framework for voice
A structured pre-production testing split looks like this: 40% happy path scenarios, 30% edge cases, 15% error handling, 10% adversarial inputs, 5% acoustic variations. Automate 80% of evaluation. Reserve human review for edge cases and novel failures.
One finding from production data stands out as a warning: small prompt changes cause large quality swings in voice agents. Every prompt update should trigger a full test run before deployment. And every production failure should become a permanent regression test case, not a one-off fix.
The trade-off to keep in mind: a perfectly accurate agent that responds in five seconds produces worse business outcomes than a slightly less accurate agent responding in one second. Your evaluation framework needs to reflect that.
EvalSpec does not measure audio, latency or speech quality. It can score the text responses behind a voice agent across five quality dimensions, and compare runs of the same system over time.
Try EvalSpec free - evaluate your AI system across all 5 dimensions. No code required.
Start your free evaluation