In March 2026, OpenAI announced it was acquiring Promptfoo, a small AI security and evaluation startup, and building its red-teaming tools straight into its agent platform. That's a significant marker. The best-funded AI lab in the world decided that testing agents for security and correctness isn't something to bolt on later; it's core infrastructure. Most QA teams are still testing agents the way they'd test a chatbot: check the final answer, move on. That gap is now the biggest blind spot in agentic AI.
The answer can be right and the process still broken
An AI agent doesn't just produce a reply. It plans steps, calls tools, retrieves information and reasons through several stages before it gives you anything to check. LangChain, the company behind widely used agent-building tools, argues testing should follow that whole path, not just the destination. It calls this trajectory evaluation: scoring the full sequence of decisions an agent makes across three areas. Whether it grounded its answer in the right information. Whether the interaction actually served the user's goal. Whether it stayed inside safety and security boundaries along the way.
Here's why that matters. An agent can reach a correct-looking answer by calling the wrong tool, retrieving the wrong document or taking a shortcut that happens to work this time and fails the next. Output-only testing can't see any of that. It only sees that the answer passed.
Test agents like distributed systems, not chatbots
A recent research paper from a twelve-author team makes a related point from a different angle. Most agent testing today is ad hoc: a handful of manually written prompts, checked by eye. The paper argues for structural testing instead, borrowing directly from decades of software engineering practice. That means building reproducible test harnesses, tracking an agent's state across multiple turns and validating every tool call rather than trusting that it worked.
The practical shift here is one of mindset. An agent behaves more like a distributed system than a script. It holds state, it calls services it doesn't fully control and it can fail in ways that only surface under specific sequences of events. Testing it like a distributed system, not a single function, is what catches those failures before a user does.
Security testing is no longer a bolt-on
The Promptfoo acquisition fits both points. OpenAI isn't just buying a red-teaming tool. It's embedding automated security and evaluation testing directly into its agent platform, so every agent built on it gets tested for vulnerabilities as standard, not as an optional extra a customer has to configure. For QA leads, that's a strong signal about where the baseline is heading. Security and correctness testing for agents won't stay a specialist add-on for much longer.
If your testing stops at whether the agent got the right answer, you're only checking the destination, not the journey. EvalSpec does not evaluate tool calls or traces today. It scores the final responses your system gives across five quality dimensions, which is one part of the picture this article describes.
Try EvalSpec free - evaluate your AI system across all 5 dimensions. No code required.
Start your free evaluation