How to Detect AI Agent Failures Automatically from Traces

To detect AI agent failures automatically, stop looking for errors and start comparing runs. Group your production traces by the job the user was trying to get done, label each run as succeeded or failed using the signals you already have, and then look for what the failing runs do that the successful ones do not. Deterministic checks catch the obvious breakages, an LLM judge catches the semantic ones, and cohort comparison tells you which of those failures actually matter.

That is the short answer. The rest of this guide explains why each piece is necessary, what each detection method misses, and how to turn a pile of traces into a short list of failures worth fixing.

Why your error rate says everything is fine

Most agent failures do not throw exceptions. The model returns a well-formed answer, every tool call gets a 200, and the run finishes in a reasonable time. Your dashboards stay green.

Meanwhile the agent told a customer their booking was changed when it never called the booking API. Or it retried the same search six times and gave up. Or it answered a refund question with the baggage policy. None of these show up as errors, because from the system’s point of view nothing went wrong.

In our analysis of the public τ²-bench airline benchmark, 22% of runs told the customer the job was done without ever calling the API that would have done it. We wrote about that in how to evaluate AI agents in production. The final message looked perfect. Only the trace showed what actually happened.

The four ways to detect agent failures, and what each one misses

There are four broad approaches. Teams usually start with the first and slowly discover they need the others.

Method	What it catches	What it misses
Deterministic checks	Tool errors, schema violations, timeouts, loops past a step limit, required calls that never happened, policy limits that were crossed.	Anything that requires understanding language: a wrong answer, an unhelpful one, or a correct action taken for the wrong request.
Statistical anomaly detection	Sudden changes in latency, token use, cost, step count, or tool mix after a deploy or a shift in traffic.	Failures that have always been there. If 15% of refund runs have been quietly failing since launch, nothing about them looks anomalous.
LLM judge on each trace	Semantic failures: hallucinated claims, ignored instructions, a skipped verification step, an answer that does not address the request.	Patterns across runs. A judge sees one trace at a time and cannot tell you whether a failure is rare or happening to a third of your users.
Cohort comparison	The decisions that separate failing runs from successful runs of the same job, and how many runs each one affects.	Very rare one-off failures, and anything you have not labeled an outcome for.

None of these is enough on its own. Deterministic checks are cheap and precise but narrow. A judge is broad but noisy and expensive. Cohort comparison is what turns individual detections into a pattern you can prioritize. The rest of this guide combines them.

A step-by-step procedure that works on real traces

Trace signals that point to specific failures

You do not need to read every trace to know where to look. These are the signals we find most often behind real failures, and the failure each one usually indicates.

Signal in the trace	Usually means
A success message with no matching write or API call	The agent claimed it finished work it never did.
The same tool called repeatedly with the same arguments	A retry loop without a change in strategy, often caused by an unhelpful tool error.
A tool error followed by a confident answer	The agent ignored the failure and filled the gap with a guess.
Input tokens growing faster in failing runs than in successful ones	Context bloat: oversized tool results or repeated history drowning the instructions. See context bloat.
The agent asking the user for something it could have looked up	A missing tool, a poorly described tool, or missing context.
A sub-agent returning far more data than the parent uses	A hand-off that adds tokens and latency without improving the answer.
A policy-relevant step missing before a write	A skipped verification or confirmation step, which is a safety issue as well as a quality one.

Be honest about what automation can do

It would be convenient if you could hand a long trace to a strong model and ask it what went wrong. The research says you cannot, at least not reliably.

The TRAIL benchmark gave long-context models 148 human-annotated agent traces and asked them to locate the errors. The best model, Gemini 2.5 Pro, scored 11%. The Who&When study of 127 multi-agent systems found that the best automated method identified the responsible agent 53.5% of the time, but pinpointed the failing step only 14.2% of the time. Even strong reasoning models did not reach practical usability.

That does not mean automatic detection is hopeless. It means a single model reading a single trace is the wrong unit of analysis. Detection gets much more reliable when you narrow the question (one workflow, one rubric item), let code answer what code can answer, and look for differences across hundreds of runs instead of trying to find the needle in one.

It also means detection costs something. Reading traces carefully, judging them against specific rubrics, and clustering cohorts takes more model calls than sampling a few runs with a generic prompt. The cheap version gives you a number. The thorough version gives you a cause. Papaya is deliberately built for the thorough version, spending the model calls needed to analyze each use case in depth.

Worked example: a travel support agent

Where Papaya fits

Papaya does this procedure for you. It reads your traces in whatever format you already produce, classifies every run into the business use case it served, and splits each use case into succeeding and failing cohorts. Its 200+ research-backed analyses then look for the patterns that separate them, from missing tool calls to context bloat to sub-agents returning too much data. You get the top issues ranked by quantified impact, not a dashboard to dig through, and with code access Papaya can open the pull request for the fix. See how it works.

Frequently asked questions

How do you detect AI agent failures automatically?

Classify production traces by the job each run was doing, label outcomes using signals like final state, user feedback, and escalations, run deterministic checks for tool and policy failures, use an LLM judge with a workflow-specific rubric for semantic failures, and compare failing runs with successful runs of the same job to find the decisions that separate them.

Why don’t AI agent failures show up in error rates?

Most agent failures are wrong decisions rather than exceptions. The model returns a well-formed answer and every tool call succeeds, but the agent skipped a step, called the wrong tool, ignored a tool error, or claimed work it never did. Only the trace shows the problem.

Can an LLM find the failure in an agent trace by itself?

Not reliably. On the TRAIL benchmark the best long-context model scored 11% at locating errors in agent traces, and on Who&When the best method pinpointed the failing step only 14.2% of the time. Narrow rubrics, deterministic checks, and comparisons across many runs are far more reliable than asking one model to read one trace.

Do I need labeled data to detect agent failures?

No. Start with outcome signals you already have, such as final database or ticket state, repeat requests, thumbs-down feedback, escalations, and reopened tickets. Where no signal exists, a workflow-specific judge can provide provisional labels that you calibrate against a small set of human reviews.

What is the most common AI agent failure in production?

It varies by agent, but the patterns we see most often are claiming success without doing the work, retry loops without a change in strategy, ignoring tool errors, context bloat from oversized tool results, and skipped verification steps. Our field guide to AI agent failure patterns covers each one.

Which tools detect AI agent failures automatically?

Papaya detects failures automatically per business use case, comparing succeeding and failing runs with 200+ research-backed analyses and ranking findings by impact. Other tools with automatic detection include LangSmith Engine, Arize Signal, Braintrust Topics, Laminar, Latitude, and Raindrop. They differ in whether detection is scheduled or on request, whether you write the detectors yourself, and whether they propose fixes.

Sources used in this article

• TRAIL: Trace Reasoning and Agentic Issue Localization — 148 human-annotated agent traces; the best model scored 11% at locating errors.
• Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems — the Who&When dataset; 53.5% agent-level and 14.2% step-level attribution accuracy for the best method.
• Why Do Multi-Agent LLM Systems Fail? — the MAST failure taxonomy, built from 1,642 annotated traces across seven frameworks.
• τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment — the public airline benchmark behind our 22% false-completion finding.