TL;DR: Traces tell you whether your tool calls succeeded, but not whether your users actually got what they came for. We built a user value loop for the Atlan MCP server to answer that question. Here’s how it works, what it told us, and what we still can’t see.
Traces are not enough
The data team at one of the world’s largest automobile companies asked their AI agent whether a specific table existed in their Atlan catalog. Our MCP server returned a clean 200 OK response, along with an empty result set.
Their next message was: “So you can’t find if that table exists?”
Nothing in the corresponding trace looked like a red flag. The agent selected a reasonable MCP tool, used valid arguments for its tool call, and our MCP server returned a successful 200 OK response quickly. However, the user walked away without an answer to a question our product could have answered.
That gap – between a successful tool call and a user who got what they came for – is the hardest thing to see when you run an MCP server. The latter is the only thing that matters to us at the end of the day, so we’ve been working to bridge that gap.
Why measuring user value for MCP servers is difficult
Measuring user value for MCP servers is a difficult problem because of the (now-stateless) nature of MCP and the variability of information we’re able to get from MCP server traces.
You never see the user
With MCP, an agent decides which of your tools to call and how – based on a prompt you’ll probably never see – and the agent then bases its response to the user on its own interpretation of your MCP server’s output. If you run an MCP server, everything you instrument sits downstream of a decision you didn’t make and upstream of an answer you never read.
Tool design increases the odds of a better outcome, not user value
Because the agent, not the user, decides which tool to call, tool design has become an important optimisation area for MCP servers – including names, descriptions, and guidance that tells clients how, when, and when not to use their tools – to maximise the chance that clients choose the right tool from their MCP server for the task that users are trying to accomplish. Tool design is a rigorous, iterative task, but it only improves the odds of a good outcome – not whether the user actually got one.
Note that tool design only covers one half of the user experience with MCP servers. Once our server sends a response, there’s no way to guarantee the agent itself uses your response well.
The signal you get from traces is thin and partly fictional
Here’s a single trace from our server, redacted:
{
"tool": "<the tool the agent called>",
"client": "<the client it called from>",
"arguments": {
"user_query": "<the ask, as the client chose to send it>",
"rationale": "<the agent's stated reason for this call>",
"...": "<...>"
},
"response": "<what your server returned>",
"status": 200
}Two key things about this trace make it difficult for us to measure user value.
-
The user’s actual prompt isn’t in there. Many clients pass some form of the user’s prompt – e.g., a dedicated
user_queryparameter, a message-shaped input, or one embedded in the tool arguments. However,user_queryis whatever the client chooses to send – sometimes it’s the user’s raw question, and other times it’s the client’s interpretation of the question, which means you may be grading against a paraphrase rather than the customer’s words (and you can’t tell which from your side of the boundary). On top of all this, not all clients send this information at all – approximately a third of our conversations carry no information about the user’s ask, which makes them ungradable on the one question we care about. -
Nothing in the trace identifies a conversation. A single conversation can span multiple turns, and a single prompt can spawn multiple tool calls. However, MCP doesn’t give your server any reliable conversation identity across calls, and the 2026-07-28 spec revision goes further by making MCP fully stateless so every request stands alone. You could approximate a conversation by grouping calls on a combination of OAuth ID, client, and time window, but simultaneous conversations with the same agent on different topics could pollute this approach.
The measurable thing in the trace is not the thing that matters to us
Tool-call success is easy to measure – you either received a valid call and returned a valid response, or you didn’t. Whether the user got what they came for is a different and more difficult question to answer. The empty catalog result at the top of this post is a good example of this – we got a perfect score on the measurable thing in the trace (the 200 OK response), but failed the user on the thing that mattered (user value).
Far less attention gets spent on the user’s side of this, and we’ve heard the same from other MCP builders and AI platforms. We don’t think anyone believes tool-call success is the point, but it’s the clearest number that comes from our traces, so it’s become an easy metric for MCP teams to focus on.
So – how do we approach the overall problem of measuring user value for our MCP server if we never see the user and traces contain limited information?
The user value loop for MCP servers
Despite the limitations that we have to face, we’re able to measure user value for our MCP server by implementing an adapted version of the user value loop, which Prathmesh Patel and the team at MCPJam introduced us to.
Simply put, the user value loop asks one question of an MCP server: did the user get what they came for?
It aims to answer this by following the user through six stages:
- Connection: can the customer reach your server at all?
- Discovery: does the agent know your tools exist?
- Tool selection: did the agent pick the right one?
- Tool call: were the arguments valid, and faithful to what the user meant?
- Tool response: did the tool return something useful, fast, and honest?
- User value: did the user get what they came for?
The user only gets value if they pass through all six stages of the loop.
Everything below runs on recorded traces of our server’s traffic, which we capture through Braintrust. For the framework in full, read Effective MCP, Part 1: The User Value Problem.
Scoping observable stages of the value loop
Measuring value for our MCP server isn’t just about evaluating the tool call in the middle – we have to follow the user’s question through the whole execution flow, from prompt to response. However, this is an inherently difficult problem, as we can really only observe the middle stages of the loop. Connection and discovery happen before a request reaches us, and we don’t see the final answer the agent gives to the user from our response, which means we have to adapt our value loop to the following:
Using goal tagging to define a conversation
The user value loop essentially aims to answer: did our MCP server’s answer satisfy the user’s ask?
To do this, we first have to define a conversation, as a user’s interaction with our MCP server to answer their question may span multiple turns.
Given each conversation with an MCP server aims to achieve a specific user outcome, we define a conversation as a set of traces with the same outcome – specifically, the same outcome_id. This method to approximate conversations is also known as goal tagging. Specifically – we’ve added an optional outcome_id parameter to every tool in our MCP server, and then ask clients to submit it for tool calls such that when a new user goal begins, the client creates an ID that represents the goal (a short description plus a fingerprint, e.g., find-pii-columns-12ab) and reuses it for every call serving that goal. This is the exact shape of state the new spec encourages (explicit handles passed as ordinary tool arguments).
Using outcomes to define conversations allows us to easily define measurability and satisfaction below.
Defining measurability and satisfaction for conversations
A conversation is measurable if its traces carry both sides of the comparison above: the user’s ask (the user_query parameter, or a user_query/question field embedded in the tool arguments) and the output the agent walked away with.
We then call a conversation satisfied if our MCP server’s output reasonably answered the user’s ask.
Note: The rationale field looks like a reasonable substitute for the user’s ask, but isn’t as it only explains why the agent chose the tool. If the agent misunderstood the user, the rationale simply describes the agent’s misunderstanding, not the user’s need. We validated this by sampling our traces: several showed a rationale that described a different intent than the user’s ask suggested.
Grading satisfaction
We use three types of graders to determine whether a conversation is satisfied:
Deterministic checks. The cheap question: does the output contain the thing the ask named? E.g., we don’t need a model to tell us if someone asked about customer_orders and customer_orders is in the response.
An LLM judge. If deterministic checks can’t decide, then an LLM judge grades the conversation to take care of the semantic middle. This essentially asks “could a competent agent answer the user’s ask from this output?” The judge provides a Pass, Partial, or Fail verdict, along with a written reason.
Human calibration. We hand-review a small set of conversations regularly – across all verdicts, not just the failures – and compare them with the judge’s to gradually calibrate the judge. This matters because uncalibrated judges are known to drift from human judgment. Even one pass over 50 traces showed inconsistencies with our judge and surfaced several bugs in the measurement pipeline itself, which we were able to fix – we had been reading numbers that were partly an artifact of our own code.
Determining failure mode
When a conversation is unsatisfied, we determine where it failed by walking through the following four questions in order:
| # | Question | If no |
|---|---|---|
| 1 | Could the tool the agent picked have satisfied this ask? | Selection failure |
| 2 | Did that tool get arguments that were semantically appropriate and schema-valid? | Call failure |
| 3 | Did the tool honour its contract for those arguments? | Response failure |
| 4 | Does the output answer the user’s original ask? | Value failure |
Note that some asks will turn out to be ones that none of our tools could have served. Those are cases of unmet demand rather than selection failure, so they go to the roadmap rather than to a fix queue.
Our learning: A value failure is usually a context layer failure
The distinction between the last two stages looks pedantic, but is important because the two failures have different owners.
As an example, suppose a user asks “what does the customer_orders table mean?” and the agent calls the MCP server’s search tool to find the requested table.
If the table exists but the tool returns an empty result, then that’s a response failure: the tool broke its own promise to find assets that exist, and the fix belongs to the service behind the tool.
However, if the tool returns the right table but the table’s description is blank, then that’s a value failure: the agent selected the correct tool, used valid arguments, and the server honoured its response contract, but the user still couldn’t answer their question because the content wasn’t there. Here, the fix goes to the context layer, because a value failure on our MCP server usually means the agent did everything right and reached into a catalog where the context was missing.
The presence of value failures in our own MCP server was further confirmation of Atlan’s broader thesis around why good context is essential for agents to deliver user value.
How the value loop changes what we ship
The loop only matters if it changes what (and how) you ship. With a measurable set of conversations and a grading method, we can now easily answer questions that drive action – e.g., what percentage of conversations deliver value, and at which stage in the value loop do our conversations fail to deliver value.
What we fix for each failure mode
Identifying different failure modes enabled us to prioritise improvements to different parts of our MCP server:
| Failure mode | What we fix |
|---|---|
| Selection | Tool descriptions, routing, consolidation |
| Call | Argument schemas, grounding, query generation |
| Response | The service behind the tool: errors, empty results |
| Value | The content itself: answer quality, relevance, coverage |
As an early example, we found a class of response failures where our error messages told the agent what went wrong but not what to do next. To address this, we rolled out guided errors, so every error now tells the agent what to try instead. Search tools now return a “did-you-mean” error, which enables agents to self-recover from search errors in a few seconds, rather than leaving the agent stranded.
Every confirmed failure also becomes an eval case, so the next release has to clear it – and we re-measure to check the fix actually addressed it.
Combining the user value loop with user outcomes
Raw success rates matter less than rates grouped by what customers are trying to do, so we classify conversations into the core outcomes they serve, then the jobs to be done inside each outcome. Note that this list will evolve, as classifiers will be able to identify new intents and unmet needs over time.
As an example, here’s a list of outcomes and intents for our MCP server:
| Outcome | Intent (job to be done) |
|---|---|
| Enrich | Annotate data assets |
| Consume | Search for data assets |
| Consume | Trace lineage for data assets |
| Consume | Run SQL queries on data assets |
Combining this user outcome grouping against the failure modes gives the view we actually use. At a glance, this enables us to answer two questions: whether we’re delivering value for each outcome customers come to us for, and whether each intent fails at a specific stage in the value loop or is spread evenly.
Getting better at measuring made our numbers worse
When we shipped the second version of our grading model, our satisfaction rate dropped several points. Nothing had regressed – the model simply got stricter, removed a class of false positives, and helped us be more honest with ourselves. Any team implementing a value loop should also expect their metrics to move after calibration.
The first thing it changed was us
Implementing the user value loop has been a worthwhile journey for us. It brought a user-focused view to the teams building our MCP server and conversational interfaces, and it’s helping build a culture that thinks in terms of user value rather than tool-call success. The user value loop has also become a grounding point for our user research team, which runs its own experiments on traces to dig deeper into user personas and behavioural profiles.
Learnings for teams implementing value loops
On top of changing how we ship our MCP server, there are a few key takeaways that we’d consider useful for anyone else implementing their own user value loop:
The data will never be complete (and that’s just where the industry is at)
About a third of our conversations carry no information about the user’s ask, and stitching individual tool calls into one cohesive conversation is unsolved across the industry, not just for us. Even the largest platforms and labs are working with partial pictures of their own traffic. There’s no need to wait for perfect data – we learned a great deal from an adapted loop over the subset of conversations we could measure.
Clients need to share more for user value loops to be stronger
The number of conversations we can measure is limited by what clients pass through, which is decided on the client side of the protocol boundary. One big change from client builders that would help everyone trying to measure value on the server side: pass the user’s ask through – the raw prompt from the user, not the LLM’s interpretation.
Expect to spend more time on LLM judge calibration
We also run an in-product conversational AI where we own the full loop and every turn is judge-scored. On that more mature surface, the earlier stages are largely fixed, so most of the remaining unsatisfied conversations are value failures. We expect our MCP server to converge on the same shape, but that progress comes with a catch. Specifically – given value failures are much more subjective and require LLM judging, the further we get in our MCP journey, the more time we expect to spend calibrating our LLM judge to ensure it accurately evaluates us on whether we’re delivering user value.
What’s next?
Conversations with other practitioners along the way helped us understand the state of the ecosystem. Observability platforms are moving towards what users were actually asking, the outcomes they’re trying to drive, and which fixes would change the answer, rather than focusing purely on infrastructure metrics. Nobody has completely solved this – which is part of why we wanted to document our learnings.
Recall the user from the start of this post, who asked whether a table existed and effectively got no answer. Conversations like that are no longer invisible to us – and the loop is now helping other teams at Atlan identify and prioritise the customer segments building on MCP. Keep an eye out for a blog on that soon.
Customer success is the most important thing for us at Atlan – we always want to be solving for user value, and the user value loop has helped align our teams on this mindset. If you’re also building an MCP server and working on user value, we’d love to compare notes.
Curious what people actually do with the Atlan MCP server? Check out the Atlan MCP Cookbook – a growing set of recipes for finding and understanding data, making changes safely, and keeping data trusted, filterable by use case and persona.
