Arize Phoenix

Agentic OS installs need span-level visibility into tool calls and retrieval, not only token dashboards. Phoenix is the research pick when you want self-hosted OTEL/OpenInference tracing + evals. Prefer Langfuse for a full OSS LLM-engineering platform with strong prompt/eval workflow; prefer OpenLLMetry when you only want instrumentation into an existing OTEL backend; prefer Helicone for proxy/gateway logging with minimal code change; prefer LangSmith when already deep in LangChain SaaS.

Apps emit OTEL/OpenInference traces into Phoenix. Operators inspect spans, run evals against datasets, and iterate prompts/experiments in the UI (or via MCP clients against Phoenix).

1. Deploy Phoenix (local/container/cloud).
2. Instrument the app (OpenInference / framework integrations).
3. Ingest traces; annotate and debug sessions.
4. Run datasets/experiments and LLM evals; promote changes with evidence.

Related: comet-optik · helicone · langfuse · topics/09-self-monitoring

• Use when: you need self-hosted tracing + evals with OTEL/OpenInference.
• Skip when: you only need a thin proxy logger (helicone), vendor-neutral instrumentation only (openllmetry), or LangChain-native SaaS (langsmith).
• Prefer instead: Langfuse for broader OSS LLM-ops; Opik if Comet affinity; Weave if W&B already owns experiments.

• Observability ≠ ledger: traces do not replace case/audit SoR.
• Open-core split: production managed features may push toward Arize AX.
• Instrument or see nothing: wrong exporter config yields empty UI.
• Eval cost: LLM-as-judge burns tokens and can drift without human checks.