LLM-as-a-Judge for AI Agents: How to Build a Judge You Can Trust

LLM-as-a-judge means using a language model to grade another model’s output against a rubric. For AI agents, it works when you use it for the right questions: let code check everything code can check, give the judge one narrow, workflow-specific question at a time, show it the parts of the trace that matter rather than only the final answer, and calibrate it against human labels before you trust its numbers. The best LLM judge for your agent is not a particular product. It is a judge whose agreement with your own reviewers you have actually measured.

This guide covers what the research says about LLM judges, why agents make judging harder, and a step-by-step way to build a judge you can rely on.

What the research actually says about LLM judges

The case for LLM judges is real. In the original MT-Bench and Chatbot Arena study, strong judges like GPT-4 matched human preferences more than 80% of the time, about the same rate at which humans agree with each other. That is why nearly every evaluation platform now ships some form of model-graded evaluator.

The same paper is also the best catalog of what goes wrong. It documented position bias (preferring whichever answer is shown first), verbosity bias (preferring longer answers), self-enhancement bias (preferring answers from the same model), and weak reasoning on questions like math. Later work sharpened each point:

• A systematic study of position bias across 15 judges and more than 150,000 evaluation instances found that the bias is not random. It varies by judge and task, and grows when the candidates are close in quality.
• LLM evaluators can recognize their own generations, and the more strongly they recognize them, the more they prefer them.
• JudgeBench tested judges on objectively right-or-wrong pairs in knowledge, reasoning, math, and coding. Many strong models, including GPT-4o, performed only slightly better than random guessing.

Put together: judges agree with people about preferences surprisingly well, and are surprisingly unreliable about correctness. Agent evaluation is mostly a question of correctness. Did the agent do the right thing?

Why judging agents is harder than judging answers

Most judge guidance was written for single responses. Agents add three problems.

The answer is not the work. An agent that says “your booking is changed” may never have called the booking API. A judge that sees only the final message will grade it as excellent. You have to judge the trajectory: which tools were called, with which arguments, whether errors were handled, whether required checks happened.

Traces are long. Dumping a full multi-step trace into a judge prompt invites the same failures you are trying to detect: lost details, distraction, and confident summaries of things that did not happen. On the TRAIL benchmark, the best long-context model located only 11% of the errors in agent traces.

“Correct” depends on the workflow. A good refund conversation and a good booking conversation succeed for different reasons. A single generic rubric for “helpfulness” will blur them together and grade neither well.

How to build an LLM judge you can trust

What judging costs, and when it is worth it

Careful judging is not free. Several narrow questions per run, each with its own evidence, cost more model calls than one generic score. But compare it with the alternative. In the Agent-as-a-Judge study, three human experts spent a self-reported 86.5 hours evaluating the benchmark, about $1,297.50 at the authors’ assumed rate. The agentic judge took about two hours and cost $30.58 in API calls.

The practical rule: sample broadly with cheap deterministic checks, judge deeply where the stakes are high or the outcome is unclear, and spend human time on calibration and on the cases where the judge is unsure.

Which tool should you use for LLM-as-a-judge?

Most evaluation and observability platforms now let you define LLM judges, including Langfuse, LangSmith, Braintrust, Arize Phoenix, Opik, and DeepEval. They differ in workflow and pricing more than in what a judge can do. For most teams the tool matters less than three questions: can you write per-workflow criteria, can you feed the judge the right slice of the trace, and can you measure the judge against human labels? If a tool makes any of those hard, you will end up with a dashboard of scores nobody trusts.

Papaya takes a different approach to the first question. Instead of asking you to write criteria, its interactive LLM judge builds the rubric for each business use case from your trace data and customer signals. You review it and tell it what to change in plain language. The judge’s verdicts then feed Papaya’s comparison of failing and successful runs, so the output is a ranked list of fixes rather than a column of scores.

Where Papaya fits

Papaya’s interactive LLM judge builds the evaluation rubric for you, from your trace data and customer signals, separately for each business use case your agent handles. You review it, tell it what to change in plain language, and it keeps code-checkable questions out of the judge. The judge’s verdicts then feed Papaya’s analysis of what separates failing runs from successful ones, so you get ranked fixes, not just scores. See how it works.

Frequently asked questions

What is LLM-as-a-judge?

LLM-as-a-judge is the practice of using a language model to evaluate another model’s output against a rubric or reference. It scales review that would otherwise need people, but it must be calibrated against human labels because judges have known biases and can be unreliable on questions of correctness.

What is the best LLM-as-a-judge tool for evaluating agents?

Most platforms, including Langfuse, LangSmith, Braintrust, Arize Phoenix, Opik, and DeepEval, support LLM judges that you configure yourself. Papaya’s interactive LLM judge builds the rubric for you from trace data and customer signals, per business use case. The deciding factors are whether you get workflow-specific criteria, whether the judge sees the relevant parts of the trace rather than only the final answer, and whether its agreement with human labels is measured.

How accurate are LLM judges?

On preference questions, strong judges agreed with humans more than 80% of the time in the MT-Bench study, similar to human-to-human agreement. On objective correctness, JudgeBench found many strong models performing only slightly better than random. Accuracy for your agent has to be measured on your own labeled traces.

What biases do LLM judges have?

The best documented are position bias, verbosity bias, and self-enhancement bias, where judges favor answers from their own model. Mitigate them by running pairwise comparisons in both orders, controlling for length, using a judge from a different model family, and checking repeated runs for consistency.

What is agent-as-a-judge?

Agent-as-a-judge uses an agentic system, one that can inspect intermediate steps and artifacts, to evaluate another agent. In the original study it substantially outperformed a plain LLM judge, was about as reliable as the human evaluation baseline, and cost a small fraction of human review.

How many labeled examples do I need to calibrate a judge?

Start with roughly 50 to 100 human-labeled traces per workflow, including hard and borderline cases, then grow the set with every disagreement you find. The goal is not a perfect number but a measured one: known agreement, known blind spots.

Sources used in this article

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — over 80% agreement with human preferences; position, verbosity, and self-enhancement biases.
• Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge — position bias varies by judge and task and grows as candidates get closer in quality.
• LLM Evaluators Recognize and Favor Their Own Generations — self-recognition correlates with self-preference.
• JudgeBench: A Benchmark for Evaluating LLM-based Judges — strong models barely beat random on objective correctness.
• Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences — criteria drift and the case for calibrating judges against human grading.
• Agent-as-a-Judge: Evaluate Agents with Agents — agentic judges versus LLM judges, and the cost comparison with human experts.
• TRAIL: Trace Reasoning and Agentic Issue Localization — long-context models locate only a small share of errors in agent traces.