← Articles

Claude Managed Agents · Trace analysis

Trace Analysis for Claude Managed Agents

Papaya now reads every Claude Managed Agents session, finds where your agent goes wrong, and tells you what to improve.

Niyaz Puzhikkunnath
· 7 minute read
A closed ink chamber with a looping blue agent trajectory inside, one thread escaping and being combed into a few clear strands, one ringed in orange

Papaya now supports Claude Managed Agents! If your agents run on Anthropic’s managed platform, or soon will, you can send every session to Papaya and get a ranked list of what to fix for quality, latency, and cost. Setup takes a few minutes in the Claude Console and needs no code changes.

The most common question we get from teams making the move is a fair one: if Anthropic runs the agent for us, do we still have our traces, and is there still anything to improve? The answer to both is yes. This article explains what Claude Managed Agents give you out of the box, what Papaya adds on top, and how to connect the two.

What Claude Managed Agents are

Claude Managed Agents is Anthropic’s hosted way to run agents. You define the agent once: its model, system prompt, tools, skills, and MCP servers. Anthropic stores that definition with versions, runs the agent loop for you, and gives each session its own sandbox where the agent’s tools run, such as bash, file edits, and code execution. You start sessions and stream their events, but you no longer write or host the loop yourself. For many teams that removes a lot of infrastructure.

What Anthropic gives you, and where Papaya comes in

Anthropic gives you good ways to see what your agents did. You can open any session in the Claude Console and read it step by step. And with trace export, every session can go over OpenTelemetry to a destination you choose, with the system prompt, user messages, the agent’s replies, and every tool call and result.

So you have your traces. The hard part is reading them. Once your agent handles thousands of sessions a week, nobody on the team can go through them all, and the questions that matter are about all of them together: what are people actually using the agent for, which of those jobs keep failing, why, and what should we change first? That is the part Papaya does.

What Papaya does with your sessions

Papaya is an agent improvement platform. You send it your agent’s traces, and it works out how the agent behaves, where it fails, and what to change, without you configuring anything. Here is what that looks like once your Managed Agents sessions arrive.

Your sessions become business use cases

A production agent does many jobs at once, and each job succeeds or fails for its own reasons. So the first thing Papaya does is sort your sessions into the business use cases they served. You see how much traffic each use case takes, how well each one works, and which ones need attention. You do not tag anything. Papaya reads the conversations and works it out.

An airline support agent's traces sorted into use cases such as flight cancellation, flight change, class upgrade, and new flight booking, each with trace volume, quality, and improvement counts. Flight cancellation is flagged as needing attention.
Use cases Sessions sorted into use cases, with the one that is struggling flagged.

This is the part you will not find anywhere else. Generic dashboards tell you an error rate. Papaya tells you that cancellations are failing while bookings are fine, which is something you can act on.

Improvements, ranked by impact

Within each use case, Papaya compares the sessions that went well with the ones that did not, using 200+ research-backed analyses. What it finds becomes a list of improvements: a tool that keeps failing and gets retried, a step where the agent skips checking its own work, a system prompt rule the agent keeps misreading, a large model doing an easy step, context that grows until the agent loses track. Each improvement comes with a priority, its expected impact on quality, latency, and cost, the change to make, and the sessions that show the problem.

Papaya also finds user signals: things people try to do with your agent that it cannot do today, how often they come up, and how those conversations ended. These are not bugs in the agent. They are decisions for your product team about what to build next.

Two columns. Improvements lists prioritized fixes to the current agent with expected impact on quality, latency, and cost. User signals lists things customers tried to do that the agent does not support, with the share of conversations and how they ended.
Output Improvements fix the agent you have. User signals tell you what to build next.

A trace viewer people can read

When you want to see a session for yourself, Papaya’s summary view shows the run as a goal and a few plain-language steps, with the step where it went wrong marked. The detail view shows the full session underneath: every message, and every tool call with its request and response. A product manager can read it as easily as the engineer who built the agent.

Papaya's trace viewer showing one trace in the summary view, with the failing step selected, then in the detail view: the customer's message, tool calls with request and response, and the agent's reply claiming a refund that was never issued.
Trace viewer Summary view for the shape of the run, detail view for the exact step.

Fixes come to you

Papaya posts new high-impact improvements to the Slack channels you choose, with the problem, the fix, and the evidence. If you connect the repository where your agent’s definition lives, Papaya can open a pull request with the change, so your part is reviewing and approving it.

A Slack message from Papaya announces a new high-impact improvement for flight cancellation with the problem, the fix, and buttons to view the pull request or open the fix in Claude Code or Cursor. Beside it, a pull request opened by Papaya is ready for your review and approval.
Workflow From a Slack alert to a pull request ready for your review.

Why this fits managed agents especially well

When Anthropic runs the loop, the code you own gets smaller. What is left is the agent’s definition: the system prompt, the tools and their descriptions, the skills and MCP servers it can use, and the model and effort you choose. Those are exactly the things Papaya’s improvements change. A recommendation like “this tool description makes the agent call it with the wrong ID” or “this use case does not need the largest model” maps straight onto a field in your agent definition.

It also matters for cost and speed. Managed sessions can run long, with many tool calls in one sandbox. Papaya shows where that time and those tokens go, use case by use case, and which changes would cut them without hurting quality.

Reading sessions yourself vs. Papaya

QuestionReading sessions yourselfWith Papaya
What happened in this session?Open it and read it step by step.A summary view of the run, with the failing step marked, plus the full detail.
What do people use the agent for?Read enough sessions to work it out.Every session sorted into business use cases, with traffic and quality for each.
Which failures keep happening?Spot them in the sessions you happen to read.Repeated failures grouped, with how often each happens and the sessions behind it.
What should we change first?Weigh the problems you found by hand.Improvements to the prompt, tools and model choices, ranked by impact on quality, latency, and cost.
What are users missing?Notice the requests the agent turned down.User signals: requests the agent cannot handle today, with how those conversations ended.
How do we hear about it?Someone has to go and look.Slack alerts for new improvements, and pull requests when your repository is connected.

Setup: a few minutes, no code changes

Connecting is two short steps, one in Papaya and one in the Claude Console:

  1. Create a Papaya SDK key. In the Papaya app, open SDK Keys, create a key for your agent, and set Capture to Full.
  2. Add Papaya as a telemetry destination. In the Claude Console, open Settings → OpenTelemetry, add a Custom destination with Papaya’s endpoint and your key, choose Full content, and pick the workspaces to send.

Click Test, run one short session, and you are connected. From then on every new session reaches Papaya as it happens. If you already have sessions from before, we can pull those through the Claude API too. The full steps, with the endpoint and header to copy, are in the Claude Managed Agents setup guide.

Try it on your own sessions

Papaya’s free plan covers one agent and 1,000 traces a month, enough to see what it finds in a real workflow. Follow the setup guide, or book a demo and we will set it up with you.

Frequently asked questions

Does Papaya support Claude Managed Agents?

Yes. Claude Managed Agents can export every session over OpenTelemetry, and you add Papaya as the destination in the Claude Console. Papaya then sorts your sessions into use cases and finds improvements and user signals, the same way it does for agents you host yourself.

Can I still get traces when Anthropic runs the agent?

Yes. Trace export sends each session to a destination you choose, including the system prompt, user messages, the agent’s replies, and its tool calls and results. You can read single sessions in the Claude Console, and send all of them to Papaya for analysis.

Do I need to change my agent’s code to use Papaya?

No. Setup happens in the Papaya app and the Claude Console: create a Papaya SDK key, then add Papaya as an OpenTelemetry destination with Full content. Your agents keep running as they are.

What does Papaya add on top of the Claude Console?

Papaya reads all of your sessions together, which nobody on a team has time to do by hand. It finds the business use cases your agent serves, groups the failures that keep repeating, and recommends ranked changes to the prompt, tools and model choices, with the sessions that show each problem.

What about sessions that ran before I connected Papaya?

Trace export sends new sessions only. We can pull your earlier sessions through the Claude API, so the first analysis can start from your existing history.

Related