It’s 3 AM and a pager goes off. Error rates are climbing in one of the many customer environments we run. The clock is already running, and most of the first half hour does not go to fixing. It goes to finding out: which tenant, which service, what changed recently, is this a new problem or the same flapping alert as last Tuesday, is it us or something upstream.
That gap, between an alert firing and a human actually understanding it, is the most expensive and least interesting part of running production software. At Atlan we run a multi-tenant platform across 500+ customer environments, and our own numbers said the gap was where the time went: P90 time-to-mitigate ran into hours on a bad week, against a target measured in minutes for the availability our customers expect. The slow part was rarely the fix. It was the investigation.
So we built an agent to do the investigation. We call it Sherlock. This is the story of how it came to be, told honestly: the two agents that failed before it, the two weeks it spent burning our API budget on noise, and the discovery that shaped everything after: the agent itself was the easy part. The real work was everything we had to build around it.
A second, far more tightly gated agent acts on what Sherlock finds. It gets Part 2 to itself.
Loop engineering names the easy part
The pattern Sherlock runs on has a name now: loop engineering. You stop hand-writing every prompt and instead design the system that keeps prompting the agent for you.
The simplest definition we’ve found is this: an agent repeats cycles of work until a stop condition is met. That sounds almost too small. It is not. The hard part is deciding what starts the cycle, what counts as done, what tools the agent can reach for, and who checks the result. Claude’s own loop guide breaks loops down along those exact axes: trigger, stop condition, primitive, and task fit. Addy Osmani and LangChain have also written the broader concept up well.
We had been running an agent this way since early 2026, before the term was common. So this is not a post about discovering the pattern. It is the more useful story: the named part turned out to be the part a strong engineer can build quickly. Surviving a multi-tenant production environment took everything around it.
The primitives are easy to name and hard to make real
Before the story, the vocabulary. Strip away the product names and every useful loop has the same shape:
- Trigger: what starts the loop. A user prompt, a schedule, an external event, or a standing routine. For Sherlock, apart from an occasional outage or a human manually asking it to investigate, the trigger is usually a production alert - and not every alert deserves a model call.
- Workers: the work repeated each cycle. Form a hypothesis, choose one to three tools, read the result, update the hypothesis.
- Evidence: what the workers are allowed to inspect. Metrics, logs, Kubernetes events, recent deploys, and prior investigations.
- Validators: the stop check. What decides the work is done: a confidence threshold, a turn cap, a deterministic fallback, or a human review.
- Traces: what survives the run. The record of every step, the conclusion, the human correction, and the mistakes we explicitly do not want repeated. Traces feed back as context for the next run, and that arrow is what makes this a loop rather than a script.
This is where the “loop engineering” conversation often gets too clean. Naming these primitives is straightforward. Making them survive production is not.
In our case, the trigger needed a cheap admission filter before it. The workers needed both an agentic path and a deterministic escape hatch. The validators needed to admit that confidence is not accuracy. The traces had to include not just successful root causes, but wrong hypotheses marked with “do not repeat this.”
Three attempts to get there
Sherlock is our third investigation agent, not our first. The difficulty was never where we expected it to be.
The metrics-only agent (2025)
Our first attempt auto-joined incident channels and pulled a shared snapshot of service health. It was metrics-only, with no logs and no events, so it could tell you that something was wrong but rarely why. It was bounded by what the models of the day could do, and it quietly became furniture: present in every incident channel, load-bearing in none.
The CLI investigator
The second attempt was a command-line agent with read-only tools and domain skills. It was genuinely more capable: it could read logs, walk dependencies, check recent changes. But it ran every step sequentially, one round-trip at a time, and usually took 10+ minutes per investigation, occasionally up to 30: slower than the humans it was meant to help. It was not agent-native by design. It was a script wearing an agent’s clothes.
The agent-native rewrite
The jump came when we rebuilt it on the Claude Agent SDK in Python, running the model in-process and firing tool calls in parallel rather than one at a time. The happy path demoed beautifully on day one. But calling it a loop would have been generous. What we had was a fast agent. The parts that make the loop everyone now talks about, guardrails around the runs and traces feeding back into the next run’s context, did not exist yet; they got built only after production showed us they were missing.
So we did the obvious thing. We pointed it at production, told it to auto-investigate every alert routed into the production alert channels, and watched it fail.
Two weeks to fail in production
It took about two weeks for production to make its point.
Roughly 11,000 alerts a month reach the channels Sherlock watches. That is the full alert stream, not just the critical incidents, and the overwhelming majority self-resolve before a human finishes reading them. The rewrite investigated all of them anyway. It spent the same on a flapping known-noise alert as it did on a real customer-facing incident, and it ran straight through our API budget doing it. We turned the auto-trigger off and sat down with the traces.
Three gaps stared back at us:
- Cost scaled with alert volume, not incident complexity. Every alert got a full investigation, so the bill tracked the noisiest thing in the system.
- It re-investigated the same alert from scratch every time it fired. No memory. The hundredth occurrence cost exactly as much as the first and learned nothing from the ninety-nine before it.
- A confident-sounding wrong answer looked exactly like a correct one. There was no signal, in the moment, to tell them apart.
Look at those through the primitives and the diagnosis writes itself: the workers were fine. What was missing was everything around them. Nothing guarded the trigger. No traces fed back into context. No validator stood apart from the model grading its own homework.
The work ahead was not “make the agent smarter.” It was three things around it: a filter before the agent to stop spending on noise, memory inside it so it stopped relearning the same incidents, and cost discipline on every run so we could afford to investigate everything that mattered.
Guarding the trigger: an admission filter that never calls a model
The cheapest investigation is the one you correctly decide not to run.
Before any model executes, every alert hits a non-LLM admission filter we call the pre-filter. This is deliberately not an agent and not a model. It’s an ordered set of rules, first match wins, and it produces one of three decisions: suppress (known noise), delay (let a self-resolving blip clear, then re-check), or investigate. It’s plain code because it has to be cheap and predictable: it runs on 100% of alert volume, and the whole point is to keep the expensive part, the model, away from the noise.
The rules that do most of the work are unglamorous. A table in Postgres holds per-tenant and per-alert classifications (noise / defer / investigate), so trial tenants and known-noisy alert types are suppressed outright. Then flapping de-duplication runs in tiers: if the same (tenant, alert) pair fired in the last 10 minutes, suppress it. If it fired in the last few hours and the previous occurrence auto-resolved or came back low-confidence, suppress it. If a recent occurrence already produced a high-confidence root cause that nobody marked wrong, suppress the recurrence and point at the existing finding.
Every decision the filter makes is logged. That log is not exhaust; it’s the training signal for the learning loop (more on that in Part 2). The filter suppresses over 85% of the alerts that reach it before a single token is spent. That one number is what turns “investigate everything” from a budget fire into a system that ties spend to incident complexity instead of alert volume.
Suppression is a stopgap, though. The better fix is fewer noisy alerts in the first place - driving that raw volume down at the source is a goal of its own, and a story for a later post.
Inside the investigation: a story in three modes
The investigation itself went through its own arc, and it’s worth telling as one, because we suspect most teams will walk the same road.
We started deterministic
The first production-worthy version of Sherlock ran a linear, deterministic workflow. It followed a fixed sequence of phases: confirm the tenant exists and resolve its context, batch-fire a wave of metric queries for global health, scope down to the affected cluster and node, then go deep into logs, recent code changes, and prior incidents.
It was predictable and thorough, which is what you want when the same rocks need to be turned over every time. It was also rigid. Every incident paid for every phase, whether it needed it or not, and a workflow can only find what its phases were written to look for.
Then we let it think
At the opposite end is agentic mode, the shape everyone pictures. Sherlock forms a falsifiable hypothesis, picks one to three tools, records what it found in a structured think step (hypothesis, confidence, evidence for, evidence against, what it ruled out), and decides whether to conclude or keep going.
When it works, it is dramatically cheaper and faster than the fixed workflow. An easy incident concludes in a couple of think cycles. When it does not work, it can circle a wrong hypothesis with great efficiency.
Hybrid is where we live today
Our default now is hybrid mode: start agentic and escalate. If Sherlock is still below roughly 50% confidence after three think cycles, it stops improvising, runs the rest of the deterministic workflow in plain Python (no model), and does a second pass over the combined evidence.
Cheap when the problem is easy. Thorough when it is not.
The trajectory, as models and our memory layer improve, is that fewer runs hit the fallback. The system is drifting toward pure agentic behavior, but it earns that drift run by run instead of assuming it.
What keeps the hypothesis step from improvising badly is memory: the traces primitive made concrete. At investigation time, Sherlock is handed context built from this tenant’s past investigations: human corrections first as ground-truth overrides, then a few confirmed past root causes, and, this is the part that matters, a couple of investigations we got wrong, explicitly flagged do not repeat this.
It is runtime context, not fine-tuning. Sherlock stops starting from zero, and stops confidently walking back into the same wrong answer.
Engineering each run to be cheap
The remaining discipline is cost, and we treat it as a feature, not a footnote. Three levers did most of the work.
Route the model to the work. A fast, cheap model does the grunt work; a strong general model (Claude Sonnet 4.6) handles the main investigation; the most capable model is reserved for the highest-severity, customer-facing incidents. You do not pay frontier prices to triage a pod restart.
Batch the tool calls. Instead of paying a round-trip per query, a single tool call fires dozens of metric queries in parallel in-process. One of our investigation phases issues around two dozen queries in a single batch.
Shrink the tool-call footprint. One wide query through a federated query gateway replaces what used to be many narrow ones, cutting both latency and the number of expensive model turns. That gateway has a name, and it’s worth a section of its own below.
The result is the number we care most about:
An investigation that used to cost a few dollars and take 10+ minutes now costs about $0.28 and finishes in about 2 minutes on average. Over a representative 30-day window, the pre-filter screened about 11,000 alerts, suppressed roughly 86% before a model ran, and turned the remaining ~1,550 into full investigations.
Every run is traced end to end (we use Braintrust) so we can see exactly where the time and the tokens went, per incident type. That trace is also what makes the cost legible: we can answer why one class of incident costs more to investigate than another, instead of staring at a single monthly bill.
Productionizing forced two gateways: Axiom and Phantom
Two problems only showed up when the agent met production, and both got the same shape of solution: put a gateway in front of the agent. We think of these as production patterns rather than implementation details, so they get names. You will see both again in later posts.
The tool-surface problem, and Axiom. A real investigation cuts across metrics, logs, Kubernetes events, deploy history, and prior incidents. Expose each backend as its own tool and the agent pays three times: a bigger prompt, more ways to choose the wrong tool, and a model round-trip per query. Axiom is our federated query gateway: it sits in front of those backends, takes one wide question from the agent, fans it out, and returns one merged result. We’re consolidating today’s separate per-backend tools onto it - and as we do, the agent has fewer tools to juggle and fewer turns to pay for, shrinking the tool-call footprint along with latency and cost.
The credential problem, and Phantom. The moment an agent moves from demo to production, someone asks the right question: what credentials does it hold? The naive answer is to hand it API keys the way you would a script, and that is the trap. Those keys end up where the model can read them, and an agent that ingests logs and alert payloads is one prompt-injection away from being talked into leaking or misusing them. Our answer is that the agent holds none. Phantom is a credential-proxy gateway. It holds the real credentials, hands an agent only a short-lived identity, enforces a per-agent allow-list, and logs every call: identity, action, target, timestamp. The agent never sees a real key, so there is nothing in its context to steal. Sherlock is read-only by design, which keeps a heavyweight gate out of the investigation hot path. Phantom earns its keep the moment an agent can change infrastructure, which is exactly why it is load-bearing in Part 2.
Zooming out: the outer loop
There’s a second, larger loop worth naming: the one the whole system lives inside.
Sherlock is just one box: Investigate. Most “AI for incidents” demos do that one box, and it demos beautifully. The engineering is in closing the outer cycle: filtering at the front so you only spend on what matters, acting at the back (carefully; that’s Part 2), and feeding every outcome into the next pass so the system is measurably better in month three than it was in month one.
Where humans are still required
The obvious question at this point: is anyone still on call? Yes, on purpose. It is worth being precise about where and why, because this is the validator primitive meeting reality.
A human still reviews every Sherlock conclusion before it drives an action. The reason is the caveat that shaped everything in Part 2: the confidence score is not accuracy. A model’s confidence measures how clean the evidence looked: signal clarity, whether the timing lines up, whether independent signals corroborate. Those are real signals, but they measure how convincing the evidence was, not whether the conclusion was right. A familiar-looking incident can score high and still point at the wrong cause. We watched it happen, at zero cost, while Sherlock was read-only.
What we’ve minimized is the toil, not the judgment. Go back to the 3 AM scene this post opened with. The pager still goes off. What’s different is what the engineer wakes up to: not five dashboards and a blank Slack thread, but a finished investigation with the evidence and the reasoning attached. The first half hour of finding out is already done; the human’s job is to judge it. That is the honest division of labor today: the agent gathers and argues, a human validates.
Which is the whole reason the second agent, the one that can change infrastructure, never gates a single action on the model’s confidence.
What we’d hand another engineer
- The loop is the easy part. Getting an agent to run an investigation is the demo; getting it to survive ~11,000 alerts a month is the actual engineering.
- Filter before you spend. A non-LLM admission filter suppresses over 85% of the alerts that reach it before any model runs, tying cost to incident complexity instead of alert volume.
- Give the agent memory, including its own mistakes. Past wrong hypotheses, injected as runtime context, are some of the most valuable tokens in the prompt.
- Tracing is what closes the loop. Trace every run end to end, then feed what the traces teach you back into the next run's context. Without that feedback arrow you have a script, not a loop.
- Production forces gateways. A federated query gateway (Axiom) shrinks the tool surface the agent has to reason over; a credential-proxy gateway (Phantom) means no agent ever holds a key.
- Confidence is not accuracy. A model's confidence measures how clean the evidence looked, never whether the answer was right, which is why a human still reviews every conclusion and why Part 2 exists.
The thing we’d actually pass on is the scaffolding, because that, not the loop, is what let Sherlock become part of real incident response.
Part 2 goes inside the second agent: the one that can change infrastructure. We’ll cover why its safety gate has nothing to do with the model’s confidence, why it holds no credentials of its own, and how a human, not a benchmark, slowly widened what it’s allowed to do.
