Observability

Observability for agents means you can explain what happened: which prompt version ran, which tools fired, how long each step took, what it cost, and where it failed. Under Agentic it is the instrumentation plane. Without it, Evaluations has nothing honest to score and Frameworks become black boxes.

Technically you want OpenTelemetry-shaped traces, prompt/version tags, token+dollar metrics, and PII-safe redaction. Failure modes: dashboards that show averages while a single tenant burns budget, traces that omit tool arguments you needed for audit, and “observability” that only works in the vendor’s UI so your ledger never sees the span.

Members today:

• LangFuse and LangSmith are LLM/agent tracing and eval-adjacent platforms common in LangChain-shaped stacks.
• OpenLLMetry pushes OpenTelemetry conventions for LLM spans.
• Arize Phoenix, W&B Weave, Comet Optik, and Helicone compete on tracing, playgrounds, and gateway-side visibility.

Prefer OTEL-first paths when you already operate traces in-house; prefer full LLM platforms when prompt diffing and dataset curation are the bottleneck. Feature maps such as Cost, Latency & Operational Economics and Observability & Auditability are the bar: can finance and engineering answer “what did this agent do, and what did it cost?” from primary data?

Navigate to Evaluations when you need offline/online scorers; to LLM Gateway when the missing piece is routing and spend controls at the API edge. Stay in Observability when the missing piece is attributable traces of agent behavior.

Instrument the path you will debug at 2am: tool args (redacted), model id, prompt hash, token counts, and the case id the run belongs to. Sampling is fine for high-volume chat; do not sample away the rare path that places an order. Export to systems your ops already trust when possible, so agent traces sit beside existing APM rather than in a silo. If cost dashboards cannot answer per-tenant spend within a day, gateway and framework choices are flying blind. Observability without retention policy is also a liability — define how long prompts live.

For this research catalog the acceptance test is boring and strict: after wiring the pack, can a second person replay the same scenario and get the same measurable outcome? If not, the page is still a brochure. Document the scenario, the expected number, and the log line that proves it. That habit is what separates a deepened taxonomy cell from a landscape screenshot.

Also record what you refused: tools not connected, corpora not ingested, metrics not collected. Refusals keep the cell honest when marketing pages claim universality. A taxonomy essay that only lists winners teaches less than one that names the skip conditions next to the install path.