LLM Observability: Monitoring Agents in Production

A production API either returns a 200 with the right payload or it doesn't. A production LLM call can return a 200 with a confident, well-formatted, completely wrong answer — and nothing in your existing monitoring stack will notice. That gap is why LLM observability exists as its own discipline rather than a checkbox inside your existing APM tool.

Teams that ship LLM features quickly tend to discover this the hard way: an agent silently starts looping on a tool call, a prompt change three deploys ago quietly degraded answer quality, or a model provider's routing change doubles latency for 10% of requests with no error logged anywhere. Traditional monitoring — uptime, latency percentiles, error rates — still matters, but it answers "is the system running" when the question that actually matters is "is the system right." This piece covers what LLM observability is, why agents make it materially harder than monitoring a single model call, what to actually instrument, and where the tooling still falls short.

What LLM Observability Actually Means

Observability, in the classic systems sense, is the ability to infer a system's internal state from its external outputs — logs, metrics, and traces. LLM observability applies that same idea to systems built around language models, but adds a layer traditional observability never had to deal with: the output itself is unstructured, probabilistic, and frequently the thing you're actually trying to validate.

Three things distinguish LLM observability from conventional application monitoring:

• The "correctness" axis is fuzzy. A database query either returns the right rows or it doesn't. An LLM response can be grammatically perfect, on-topic, and factually wrong, or technically correct but useless for the task. You need a way to measure quality, not just availability.
• Cost and latency are usage-dependent, not request-dependent. A single API call's cost and duration depend on prompt length, output length, whether a cache hit occurred, and how many tool-call round trips happened — none of which is fixed per endpoint the way it is in a typical REST service.
• Failure is often silent. A model can produce a hallucinated citation, misuse a tool, or contradict an earlier turn in a conversation, and the HTTP layer will report total success. Observability has to look inside the content, not just the transport.

Put simply: LLM observability is the combination of tracing (what actually happened, step by step), evaluation (was it any good), and operational metrics (cost, latency, token usage, error rates) applied to systems where the core unit of work is a model inference rather than a deterministic function call.

The Three Pillars

Most mature LLM observability setups converge on the same three categories of signal, even when the tooling differs.

Tracing: what actually happened

A trace captures the full execution path of a request — the prompt sent, the system instructions, any retrieved context, the model's raw response, every tool call and its result, and the final output shown to the user. For a single-call system this is a flat record. For an agent, it's a tree: a top-level task can branch into sub-tasks, tool invocations, and even delegated calls to other agents, each with its own inputs, outputs, latency, and token cost.

Without a trace, debugging an agent means asking the model to explain itself after the fact — which is unreliable, since the model's explanation is itself a generated text completion, not a log of what it did. A trace is ground truth: the actual prompt that went in, the actual tokens that came back.

Evaluation: was it any good

Evaluation is the layer that answers the question traces can't: was this output correct, safe, and useful? This happens at two speeds:

1. Offline evaluation — running a fixed test set (a "golden dataset") against the current prompt/model/pipeline before you ship a change, an approach covered in more depth in our guide to AI agent evals, to catch regressions before they reach users.
2. Online evaluation — scoring live production traffic, either with automated graders (rule-based checks, semantic similarity, or a separate "judge" LLM call) or with user feedback signals (thumbs up/down, edits, regeneration requests, abandonment).

Neither replaces the other. Offline evals catch regressions before deploy; online evals catch the drift and edge cases that only show up once real users and real data hit the system.

Operational metrics: cost, latency, and reliability

This is the layer closest to traditional APM, but with LLM-specific units: tokens in, tokens out, cost per request, time-to-first-token (critical for streaming UX), total generation time, cache hit rate, and provider-side error rates (rate limits, timeouts, refusals) — the core AI agent metrics worth tracking from day one. These metrics need to be sliced by model, prompt version, and feature — not just aggregated globally — because a single product surface often calls several different models for different sub-tasks.

Why Agents Break Traditional Monitoring

A single LLM call is hard enough to monitor. An agent — the kind of system covered in our overview of AI agent development, where the model decides its own sequence of actions, calls tools, and iterates based on results — multiplies the problem in a few specific ways.

Non-determinism compounds across steps. A single model call has some output variance. An agent that takes ten sequential model-driven steps has ten opportunities for that variance to push the whole trajectory somewhere unexpected. The same input can produce a different tool-call sequence on different runs, which makes "did this work" a distribution question, not a pass/fail one.

There is no fixed call graph. Traditional distributed tracing (the kind built for microservices) assumes a mostly-stable service topology: request hits service A, which calls B and C. An agent's "call graph" is decided at runtime by the model itself — it might call zero tools, three tools, or the same tool five times in a retry loop. Your tracing has to capture an emergent graph, not a predefined one.

Failure is often a behavior, not an error. An agent that gets stuck alternating between two tool calls without making progress doesn't throw an exception — it just burns tokens and time until a step limit or budget cap kicks in. Detecting that requires watching for behavioral patterns (repeated identical tool calls, oscillating states, no forward progress across N steps), not status codes.

Multi-agent systems add a coordination layer. Once one agent can delegate to another — the pattern covered in building multi-agent systems — you need visibility not just into each agent's individual trace but into the handoffs between them — who was told what, what came back, and where a miscommunication between two agents caused a downstream failure that neither agent's individual trace makes obvious.

Cost and latency are directly tied to agent "decisions." A single wasted tool call in an agent loop isn't just wrong — it's billed. Runaway loops are both a correctness problem and a cost incident, and the two need to be visible in the same view for anyone to catch them before the bill does.

Benefits of LLM Observability

Instrumentation costs engineering time and storage. What it returns is the ability to run LLM systems with the same confidence teams expect from the rest of their stack.

Debugging From Evidence Instead of Guesswork

When a user reports a bad answer, a trace shows the exact prompt, retrieved context, tool calls, and raw output that produced it. Engineers can tell whether retrieval returned the wrong passage, a tool failed, or the model misread good context, and fix the right layer. Without traces, teams reproduce issues by trial and error or ask the model to explain itself, which produces plausible stories rather than facts. Time to diagnosis drops from days to minutes for most issues.

Catching Quality Regressions Before Users Do

Offline evaluation against a golden dataset flags when a prompt change or model swap makes answers worse on cases that used to work. Online scoring then catches drift that only appears with real traffic. Together, they turn "it feels worse lately" into a measurable signal tied to a specific deploy. Teams ship changes faster because they can see the impact rather than hoping for the best.

Cost Under Control

Token usage and cost broken down by feature, model, and prompt version show where spend actually goes. Runaway agent loops, retry storms, and context growth become visible as they happen rather than at the end of the billing cycle. That visibility supports concrete decisions, such as moving a sub-task to a cheaper model or capping steps for a particular agent, instead of across-the-board cuts.

Faster, Safer Iteration on Prompts and Models

Version tags on every trace connect quality, latency, and cost to the exact prompt and model that produced them. Teams can compare versions side by side in production, roll back with confidence, and justify switching providers with data. Prompt engineering becomes an evidence-based practice instead of a series of untracked edits.

An Audit Trail for Safety and Compliance

Logging guardrail events, refusals, and PII detections alongside full traces gives security and compliance teams a record of what the system did and why. When questions come from customers, auditors, or internal reviewers, the answer is a query rather than a reconstruction from memory.

LLM Observability Use Cases

Every LLM system benefits from some instrumentation, but the emphasis shifts with the kind of system being watched.

Customer-Facing Support Agents

Support agents answer real customers, so wrong answers cost trust immediately. Traces show which knowledge base passages were retrieved for each reply, online graders flag answers that contradict the source, and user signals such as rephrasing or escalation highlight conversations for review. Teams use this to find gaps in content, adjust escalation rules, and confirm that each prompt change improves resolution rather than just shifting where failures happen.

Retrieval-Augmented Assistants

In RAG systems, most bad answers start with poor retrieval. Observability that records retrieved chunks, their scores, and whether the final answer actually used them separates retrieval failures from generation failures. That tells the team whether to tune chunking and search or to change the prompt, rather than guessing. Over time, logged queries also reveal topics the knowledge base doesn't cover at all.

Tool-Using and Coding Agents

Agents that call APIs, run code, or modify data can fail through misuse rather than errors: looping on a tool, passing fabricated arguments, or taking more steps than needed. Tool-call outcomes, step counts, and behavioural alerts for repeated identical calls catch these patterns early. Step limits and budget caps can then be set from observed behaviour rather than arbitrary guesses.

Multi-Agent Workflows

When a coordinator delegates to specialist agents, a failure may originate in an instruction, an execution, or a misread handoff. Hierarchical traces that capture each handoff, with the exact message passed between agents, give reviewers what they need to locate the break. Without them, teams see only a wrong final output and a set of individually plausible agent logs.

Cost Governance Across Teams

Organisations with several LLM features need to know which products and teams drive spend. Cost metrics tagged by feature and team support budgeting, chargeback, and decisions about where cheaper models are good enough. Finance gets numbers it can plan around instead of a single opaque provider invoice. Product owners can also see the cost of each feature next to its usage and decide whether it earns its keep.

What to Instrument: A Practical Checklist

Teams new to this tend to either over-instrument (log everything, drown in noise) or under-instrument (log only errors, miss the quality problems). A reasonable middle ground, ordered roughly by how early you should add each one:

A few implementation notes worth calling out:

• Capture the raw prompt, not a summary of it. Redacted or summarized logs are cheaper to store but useless when you need to reproduce a bug exactly.
• Tag traces with a prompt/model version identifier from day one. Retrofitting version tracking after a quality regression means you can't tell which change caused it.
• Sample selectively, don't sample uniformly, once volume is high. Prioritize capturing traces that hit errors, took unusually long, cost unusually much, or received negative user feedback — uniform random sampling under-represents exactly the traces you need.

Building an Observability Stack

There isn't one dominant standard yet, so most teams choose between a few approaches depending on how much of the stack they want to own.

Purpose-built LLM observability platforms

Tools built specifically for this category (LangSmith, Langfuse, Helicone, Arize Phoenix, and similar products) provide trace visualization, prompt versioning, dataset-based evaluation, and cost dashboards out of the box, usually with SDK instrumentation that wraps your existing model calls. These get you moving fastest, particularly for teams already using an agent framework the tool integrates with directly.

Extending existing APM/observability tooling

Teams with an established observability vendor (Datadog, New Relic, Honeycomb, and others) increasingly have LLM-specific modules that plug into the same dashboards used for the rest of the infrastructure. This avoids fragmenting on-call workflows across multiple tools, at the cost of sometimes-shallower LLM-specific features (like built-in eval scoring) compared to purpose-built tools.

Building on OpenTelemetry

OpenTelemetry has an emerging set of semantic conventions for generative AI spans (model name, token counts, prompt/completion attributes), which lets you instrument LLM calls the same way you already instrument the rest of a distributed system and route the data to whatever backend you already use. This is the most vendor-neutral path but requires more setup work, since the conventions are still evolving and coverage varies by SDK and framework.

A rough decision framework

Whichever route you pick, treat the observability layer as part of the system's design from the start, not something bolted on before a launch. Agents that were built without any tracing hooks are expensive to retrofit, because you end up debugging blind for however long that gap lasts.

Common Failure Modes Observability Is Meant to Catch

Once instrumentation is in place, here's what it typically surfaces that would otherwise go unnoticed:

1. Prompt drift regressions — a small prompt tweak meant to fix one issue quietly degrades performance on a different task the prompt also handles.
2. Tool misuse loops — the agent calls the same tool repeatedly with slightly different arguments, never converging, until it hits a step or token limit.
3. Context window creep — a long-running conversation or agent session accumulates so much history that relevant instructions get pushed out or diluted, a dynamic explained further in context windows explained, and output quality degrades gradually rather than suddenly.
4. Silent provider degradation — a model provider changes routing, load-balances to a different underlying checkpoint, or has a partial outage that manifests as slightly worse answers rather than outright errors.
5. Cost spikes from retries — an upstream service's retry logic combined with an agent's own internal retries multiplies the actual number of model calls per user action far beyond what anyone budgeted for.
6. Hallucinated tool arguments or citations — the model calls a real tool with a plausible-looking but fabricated argument, or cites a source that doesn't say what the model claims, the same underlying failure mode covered in why AI hallucinates.
7. Multi-turn inconsistency — the agent contradicts something it said three turns earlier, which is invisible if you only ever look at single-turn traces.

Every one of these looks completely healthy from the outside if all you're watching is HTTP status codes and p99 latency.

Common LLM Observability Mistakes

Instrumentation is only half the job. These are the mistakes that leave teams with dashboards but without answers.

Relying on Traditional APM Alone

Uptime, error rates, and latency percentiles stay green while the model gives wrong answers, loops on a tool, or drifts after a provider change. Teams that stop at APM learn about quality problems from users. By then the damage to trust is done, and the evidence needed to diagnose the issue was never recorded. LLM systems need traces of content and evaluation signals on top of the usual infrastructure metrics.

Logging Summaries Instead of Raw Prompts

Storing truncated or summarised prompts saves space but makes exact reproduction impossible. When a bug depends on a particular phrase, a retrieved chunk, or an instruction buried in the system prompt, a summary hides the cause. Capture the raw request and response, and manage storage cost through retention and sampling instead. Redact sensitive fields deliberately rather than dropping whole prompts.

Shipping Without Version Tags

Without a prompt and model version on every trace, a quality drop can't be tied to the change that caused it. Teams end up bisecting deploys by hand or guessing. Adding version identifiers costs almost nothing at the start and is painful to retrofit after an incident. Include the retrieval index version too, since a re-indexed knowledge base can change answers as much as a prompt edit.

Sampling Uniformly at Scale

Random sampling keeps a representative slice of normal traffic and discards most of the rare, costly, or failed requests that matter. Prioritise traces with errors, high latency, high cost, or negative feedback so the interesting cases are always available for review. Keep a small uniform sample as well, to track overall baselines.

Trusting the Judge Model Without Checking It

LLM-as-judge scores are convenient, so teams sometimes treat them as ground truth. Judges share blind spots with the models they grade and can reward confident wrong answers. Periodically compare judge scores with human ratings on the same traces, and only trust automated scores where they agree.

LLM Observability Best Practices

• Instrument from the first prototype. Add tracing hooks while the system is small. Retrofitting tracing into a mature agent means debugging blind until the work is done. Early traces also become the first entries in your evaluation dataset.
• Capture the full trace tree. Record parent-child relationships for every model call, tool call, retrieval step, and sub-agent so the emergent call graph can be reconstructed in order. Include timing for each step so slow tools and slow models can be told apart.
• Tag every trace with prompt and model versions. Make the identifiers part of the request context so dashboards and evaluations can be sliced by version automatically.
• Maintain a golden dataset and run it before each deploy. Grow it from real production failures, so every fixed bug becomes a regression test. Review the set quarterly and retire cases that no longer reflect how people use the product.
• Score a sample of live traffic continuously. Combine automated graders with user feedback signals, and route low-scoring traces to human review. Track how often reviewers disagree with the automated score.
• Alert on behaviour, not just errors. Watch for repeated identical tool calls, rising step counts, growing context size, and sudden cost per request jumps, and set step and budget caps from observed patterns. Route these alerts to the same on-call rotation as infrastructure alerts, so they get a response.
• Slice metrics by feature, model, and version. Global averages hide the one feature or model that is degrading; breakdowns surface it. Review the breakdowns after every deploy, not only when something looks wrong.
• Design data governance alongside tracing. Decide what gets redacted, where traces are stored, and how long they are kept, in line with privacy and residency requirements, before full-fidelity logging goes live. Restrict who can read raw traces, since they often contain customer data.

Limitations and Open Questions

LLM observability is a younger discipline than the systems monitoring it borrows vocabulary from, and it has real gaps worth being honest about.

Automated evaluation is itself imperfect. Using an LLM to grade another LLM's output ("LLM-as-judge") is common and useful, but it inherits the judge model's own biases and blind spots — it can be fooled by confident-sounding wrong answers just as a human skimming quickly might be. Judge scores are a signal to investigate, not a ground truth to trust blindly.

There's no universal standard yet. Unlike HTTP status codes or standard log formats, there's no single agreed schema for what a "trace" or an "eval" looks like across tools. OpenTelemetry's GenAI semantic conventions are moving toward filling this gap, but adoption is uneven, and switching observability vendors today usually means re-instrumenting rather than just repointing an exporter.

Instrumentation has real overhead. Logging full prompts, full responses, and full tool payloads for every request adds storage cost and, for very high-volume systems, can add latency if done synchronously. Teams have to make deliberate tradeoffs about sampling and what gets captured at full fidelity versus summarized.

Attribution in multi-agent systems is genuinely hard. When three agents collaborate and the final output is wrong, tracing shows you what each agent did, but assigning "whose fault was it" — a bad instruction from the coordinator, a bad execution from the worker, or a bad interpretation of a correct instruction — often still requires human judgment.

Privacy and compliance constraints limit what you can log. Full-fidelity tracing means capturing user inputs and model outputs verbatim, which runs directly into data residency, PII handling, and retention requirements in regulated industries. Observability strategy and data governance strategy have to be designed together, not sequentially.

What to Watch Next

The space is consolidating in a few directions worth tracking if you're building on this stack long-term:

• Standardization around OpenTelemetry's GenAI conventions — as more SDKs and frameworks adopt the same span attributes for model calls, switching or combining observability backends should get cheaper.
• Built-in observability from agent frameworks — as agent orchestration frameworks mature, more of them are shipping tracing and evaluation hooks natively rather than requiring a separate SDK to be bolted on.
• Continuous/online evaluation becoming table stakes — the gap between "we ran evals before launch" and "we're scoring every production request" is closing, since offline test sets consistently under-represent real user behavior.
• Better tooling for multi-agent attribution — as multi-agent systems move from research demos to production features, expect more specialized tracing for cross-agent handoffs specifically, rather than treating each agent as an isolated black box.

Teams building or scaling LLM-powered products who want help designing this instrumentation from the ground up, whether as part of a broader LLM integration engagement or a standalone review, can reach out to Woyce Technologies.

FAQ

What is LLM observability?

LLM observability is the practice of monitoring, tracing, and evaluating LLM-powered applications in production — capturing not just uptime and latency but the actual prompts, responses, tool calls, and quality of outputs so teams can debug and improve the system after it ships. It combines three kinds of signal: traces that record every step an agent took, evaluations that score whether the output was any good, and operational metrics such as token usage, cost, and time-to-first-token, sliced by model and prompt version.

How is LLM observability different from traditional application monitoring?

Traditional monitoring focuses on availability and performance (is the service up, how fast is it responding). LLM observability adds a quality dimension — was the output correct, safe, and useful — because an LLM call can return a technically successful HTTP response that is still wrong, hallucinated, or off-task. It also tracks units that APM tools never had to care about, such as tokens, prompt versions, and tool calls chosen at runtime by the model. You still need uptime and latency monitoring; LLM observability sits on top of it rather than replacing it.

Why is monitoring AI agents harder than monitoring a single LLM call?

Agents make their own runtime decisions about which tools to call and in what order, so there's no fixed call graph to monitor against. Failures also often show up as behavior — like repeated tool calls or context drift — rather than as errors, which requires watching for patterns across a trace instead of just checking status codes.

What should I log for an LLM-powered feature?

At minimum: the full prompt and response, token usage, latency broken down by step, tool call inputs/outputs, the prompt and model version, and some evaluation signal (automated grading, user feedback, or both). Prioritize capturing traces tied to errors, high cost, or negative feedback if you can't log everything at full fidelity.

What tools are commonly used for LLM observability?

Options range from purpose-built platforms (LangSmith, Langfuse, Helicone, Arize Phoenix) to LLM-specific modules inside existing APM vendors (Datadog, Honeycomb, New Relic) to a vendor-neutral approach built on OpenTelemetry's generative AI semantic conventions. The right choice depends on how much of the stack you want to own and whether you're already invested in an existing observability vendor.

Can automated evaluation replace human review of LLM outputs?

Not entirely. LLM-as-judge scoring and rule-based graders are useful for catching regressions and scaling evaluation beyond what humans can review manually, but they inherit the judge model's own blind spots. Most reliable setups combine automated scoring with periodic human review and real user feedback signals. A practical pattern is to have humans regularly grade a sample of traces, then compare their scores with the automated judge so you know where the judge can be trusted and where it drifts.

Is OpenTelemetry a good foundation for LLM observability?

It's a reasonable choice if you want vendor neutrality and already use OpenTelemetry elsewhere in your stack. Its generative AI semantic conventions are still maturing, so coverage varies by SDK and framework, but it avoids locking your tracing data into a single proprietary format. Check that the SDKs and frameworks you actually use emit the attributes you care about, such as token counts and tool calls, before committing to it.

Conclusion

The core problem is simple to state and easy to miss: an LLM system can be fully available, fast, and error-free by every traditional metric while giving users wrong answers. Agents make it worse, because they choose their own steps at runtime, so failures show up as loops, drift, and wasted spend rather than exceptions.

The useful takeaways are practical. Treat traces as ground truth rather than asking the model to explain itself. Pair offline evals before deploys with online scoring of live traffic. Tag every trace with a prompt and model version so regressions can be tied to a change. And sample deliberately, keeping the expensive, slow, and negatively rated traces rather than a uniform slice.

The caveats are real. LLM-as-judge scores are signals, not truth. Standards are still settling, so vendor choices carry some switching cost. Full-fidelity logging collides with privacy and retention rules, which means observability and data governance need to be designed together.

A good first step is to instrument one agent end to end with full traces, version tags, and a small golden dataset, then review a week of production traces before scaling up. If you want help building that foundation into a production AI system, explore our AI agent development services.