How to Evaluate AI Agents in Production: A Practical Framework

Imagine a support agent that correctly refunds a customer. The final state looks perfect: refund issued, ticket closed, customer notified. Then you inspect the trace. The agent tried the same tool four times, exposed an internal policy in its explanation, skipped the eligibility check, and only succeeded because the final retry happened to work.

Did the agent pass?

If your evaluation checks only the final database state, yes. If you care about reliability, policy, cost, latency, or what happens on the next thousand runs, no.

And that is the flattering version, where the job at least got done. The 22% figure above comes from our analysis of the public τ²-bench airline benchmark: the agent told the customer the job was done, but no real reservation API had ever been called. The final message looked perfect. The database knew better.

An agent does more than produce an answer. It observes, makes decisions, calls tools, changes state, and sometimes recovers from errors. The evaluation has to cover those actions as well as the final result.

First, define what success looks like

Teams often begin by asking which evaluation framework to use. Before choosing one, write down what the agent is supposed to accomplish.

A useful job contract has four parts:

User goal	The real-world result the user expects, not merely the wording of the prompt.
Required state	The observable facts that must be true when the run finishes.
Constraints	Policies, permissions, sources, timing, and actions the agent must respect.
Acceptable failure	When the agent should ask, abstain, escalate, retry, or stop safely.

For a refund agent, the goal is not to write a helpful response. It is to resolve an eligible refund correctly. The required state might include a refund record and an updated ticket. The constraints include the refund window, approval limits, and which payment tool may be used. If the policy is ambiguous, the correct action may be to escalate.

Once the job is clear, the metrics become easier to choose. More importantly, you are less likely to optimize whatever happens to be easy to score.

What you actually need to measure

An agent can fail in several different places. The final result can be wrong, the path can be wasteful even when the result is right, one critical tool call can be unsafe, and a run that does everything correctly can still be too slow or too expensive for production. Those failures should not be hidden inside one score.

Keep these measurements separate. If quality falls, you need to know whether the cause was a bad retrieval, a tool failure, a poor decision, a new kind of task, or a latency timeout. A blended “agent score: 78” does not tell you.

Do not use an LLM when code can answer the question

If the ticket reopened, the test failed, or the transaction ended in the wrong state, keep that outcome. If code can verify a schema, a policy limit, a required tool call, or a database change, use code. These checks are cheap, repeatable, and easy to debug.

Human review and LLM judges are useful when the question is genuinely subjective — whether the answer was complete, whether it used the evidence correctly, whether each step actually helped. Human review is expensive, so use it for important, ambiguous, or novel cases and for a small calibration set.

LLM judges can scale that review, but they are not neutral. The original MT-Bench and Chatbot Arena study made the case that LLM judges can scale open-ended evaluation and often agree with human preferences. The catch is that the same paper documented position, verbosity, and self-enhancement biases.

A later study of position bias focuses on one of those weaknesses. Its main finding was that position bias is systematic and varies by judge and task. The practical lesson is simple: evaluate the judge, not just the agent.

There is an important limit to that research. MT-Bench studied preferences between chatbot answers. It does not show that the same judge can reliably verify tool calls, policy compliance, or the full path an agent took. Those questions need their own evidence and calibration.

What an LLM judge needs

• A concrete rubric. “Be helpful” is not a useful criterion. “Answer all three requested questions and cite the supplied policy for each claim” is.
• The right evidence and an uncertainty option. Give the judge the user goal, final state, relevant tool results, constraints, and the full conversation when earlier turns matter. If there is a known correct answer or expected state, include it. Let the judge return “needs human review” when the evidence is insufficient.
• Calibration and versioning. Compare the judge against objective or human labels. Record the model, rubric, threshold, prompt version, sample, and cost with every result.

In practice, run the same comparison more than once and swap the order of the answers. If the winner changes, record the result as uncertain instead of forcing a decision. This matters most when the answers are close in quality, which is also when a small ordering bias can decide the result.

We have not found a shortcut here. Until the judge demonstrates agreement on your workflow, label its scores as uncalibrated. It can still find suspicious runs and help build a dataset. It should not silently replace the outcomes you already trust.

Look for patterns across real runs

A strange trace shared in chat, a customer complaint, or a particularly good demo can tell you where to look. None of them tells you how often the behavior occurs.

This is where production cohorts help. A production cohort is a group of runs that share a property you care about: the same task family, workflow version, model, customer segment, language, risk level, or outcome. Instead of comparing an arbitrary successful run with an arbitrary failure, compare similar runs and look for the first meaningful difference.

Does a new prompt improve the average while hurting a small but valuable cohort? Is the cheaper model still cheaper after you include retries and failed runs? Do failures begin after the same retrieval step? These are questions you cannot answer from one screenshot.

This is also where real traces beat intuition. In the same τ²-bench analysis, one overall pass rate hid a 19-point cohort gap: runs that leaned on a free-form code sandbox passed 29% of the time, while runs that stayed on structured tools passed 48%. Inside the failing cohort, five runs burned 241 redundant calls retrying one broken tool, because nothing told the agent to stop. A blended score pointed at none of it.

Test before release, then keep checking

Before release, run a versioned set of representative and difficult cases. When possible, replay recent production traces against the proposed change without allowing real side effects. After release, sample live runs, attach the real outcomes, and watch quality, cost, latency, and failure patterns by production cohort.

Your golden set should be a small, versioned collection of cases the team understands well. It should include common work, important edge cases, and failures that have already hurt you. The golden set catches known regressions. Production monitoring catches changes you did not anticipate. You need both.

Start with one workflow

Pick one repeated production workflow and review 20 or 30 real runs by hand. Write the job contract, identify two or three production cohorts, and decide which mistakes matter most. Do not begin by designing a universal evaluation system.

Next, capture enough of the run to explain what happened: a stable run ID, workflow version, model and prompt version, tool calls and results, timing, token usage, errors, final output, and available business outcomes. The OpenTelemetry GenAI conventions exist to give agent, model, and tool activity a common trace structure. An agent span contains model and tool spans, with attributes for the model, token usage, duration, and finish reason. Full prompts and tool arguments may contain sensitive data, so content capture should be deliberate rather than automatic.

Instrumentation gives you the evidence, not the verdict. A trace can show which tools ran, how long they took, and what they returned. It cannot tell you whether the customer’s problem was solved. That still comes from outcomes, explicit contracts, and the evaluators you choose.

Add deterministic checks for the outcomes and critical steps you can verify exactly. Label a small set by hand. If important questions remain that code cannot answer, add an LLM judge and measure its agreement with that set.

Finally, freeze the first golden set and baseline. Every meaningful change should be compared against them before release and against production cohorts afterward. The system is useful when the team can tell whether the workflow improved, which users were affected, and where a new failure began.

Where evaluations go wrong

The first mistake is checking only the final answer. That misses bad tool calls, policy violations, wasted work, and lucky outcomes. The second is collapsing everything into one number. Once outcome, process, reliability, cost, and latency are blended together, the score stops explaining what changed.

A subtler one is trusting an uncalibrated judge because its rationale sounds confident. Record where every score came from. Human feedback, deterministic checks, imported labels, and model judgments are not interchangeable.

The hardest mistake to notice is changing the agent and the evaluation at the same time. If both move, an improvement is always debatable. Version the agent, the rubric, the evaluator, and the golden set.

Now return to the refund agent. Suppose the next release raises its pass rate but doubles its retries and skips the eligibility check more often. Did it improve? It did not. Your evaluation should make that answer obvious.

Frequently asked questions

What is the best way to evaluate an AI agent?

Evaluate it at four levels: outcome, trajectory, critical steps, and system behavior. This combination tells you whether the job succeeded, whether the path was reasonable, whether important actions were correct, and whether the overall run was reliable enough for production.

Which AI agent evaluation metrics should I track?

Begin with task success and outcome coverage. Add tool-call correctness, policy compliance, retry and recovery rates, latency, cost per successful run, escalation rate, and human correction rate. Always segment by workflow version and meaningful production cohort.

How many examples do I need for an agent eval?

There is no universal number. Start with 20–30 manually reviewed production runs to define the problem, then build a compact regression set that covers common, high-value, risky, and previously failed cases. Use larger production cohorts to measure real-world rates and rare failures.

Should I use an LLM as a judge?

Use an LLM judge for semantic questions deterministic code cannot answer well. Give it a workflow-specific rubric and relevant evidence, allow uncertainty, and calibrate it against objective or human labels. Do not let an uncalibrated judge silently become ground truth.

What is trajectory evaluation?

Trajectory evaluation examines the sequence of observations, decisions, model calls, tool calls, state changes, retries, and verification steps that led to an outcome. It catches brittle or unsafe behavior that final-answer evaluation misses.

Where Papaya fits

Papaya compares repeated production runs, connects outcomes to traces, and keeps deterministic checks separate from model-based judgments. The output is not another score — it is what is going wrong, which change is worth making, and whether that change worked. See how it works.

Sources used in this article

• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — informed the use of final state, policy compliance, realistic conversation changes, and repeated trials as measures of reliability.
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — informed the guidance on judge bias, answer-order checks, reference answers, and keeping the full conversation available for multi-turn evaluation.
• Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge — informed the recommendation to test a judge for repeatability and position consistency before relying on it.
• Inside the LLM Call: GenAI Observability with OpenTelemetry — informed the instrumentation guidance for agent, model, and tool spans, while keeping observability separate from evaluation.