AI agent observability tools help teams reconstruct what an agent did, measure whether the result was acceptable, and connect failures to latency, model usage, retrieval, or tool calls. The difficult part is not collecting more logs. It is preserving enough causal structure to answer useful questions without creating a new store of sensitive prompts and outputs.
For most teams, the practical choice is not “Which dashboard has the most features?” It is “Which system can capture our agent’s execution model, support our evaluation workflow, meet our data boundary, and produce costs we can reconcile?” This guide compares those requirements and gives you a pilot scorecard.
Product, pricing, retention, and compatibility statements in this article were checked against official vendor documentation on 2026-07-23. Vendor pages can change, and two noted values conflict across official pages; those items remain marked [VERIFY].
Traces are not evaluations
A trace explains what happened during one execution. An evaluation estimates how well that execution performed. You usually need both, but they solve different problems.
An agent trace should preserve the parent-child relationship among the user request, planning or orchestration steps, model calls, retrieval, tool execution, retries, and final response. A span should record one timed operation inside that trace. With those relationships intact, an engineer can locate a slow tool, a repeated model call, or a retrieval step that supplied irrelevant context.
An evaluation attaches a judgment to a trace, span, session, or experiment. The judgment might be deterministic, such as valid JSON or a successful database write. It might compare the result with a reference answer. It might be a human label, user feedback, or an LLM-as-judge score. LangSmith’s official evaluation model, for example, distinguishes offline evaluation on datasets from online evaluation on production runs and supports human, code, LLM-as-judge, and pairwise techniques (official evaluation concepts). Langfuse likewise stores human, model-based, code-based, and user-feedback judgments as scores (official scores documentation).
Do not treat a successful trace as a successful agent run. A trace can be complete while the answer is wrong. Conversely, a failed task may reveal that the agent correctly stopped at an approval gate. Define outcome metrics separately from transport and runtime health.
If you are deciding where automated checks should block a release, the existing guide to a self-hosted evaluation gate covers that control point in more detail.
Build a core telemetry model before choosing a tool
A vendor demo is easier to judge after you define the minimum record your system needs. Start with a portable schema rather than instrumenting every object exposed by one SDK.
For each trace, capture:
- a trace ID and, for multi-turn work, a session ID;
- application, environment, release, route, and tenant-safe identifiers;
- start time, end time, status, and a normalized failure class;
- model provider and model identifier actually used;
- input, output, cached, and reasoning token counts when the provider supplies them;
- estimated model cost and the price-table version used;
- tool name, attempt number, duration, result status, and approval outcome;
- retrieval collection, document-safe identifiers, result count, and timing;
- evaluation name, version, score, explanation, and evaluator cost;
- sampling decision and redaction-policy version.
Names matter. If one service records a web search as search, another as tool.search, and a third as browser, cross-version dashboards and evaluators become unreliable. Establish a small taxonomy for span type, error class, tool outcome, and agent version. Keep volatile details in attributes rather than embedding them in span names.
OpenTelemetry can reduce lock-in, but “supports OpenTelemetry” is not a complete compatibility test. Check whether a product accepts standard OTLP, which generative-AI attributes it maps, whether it preserves custom attributes and links, and whether you can export the resulting data again. LangSmith documents OTLP ingestion and its mapping of OpenTelemetry and GenAI attributes (official OpenTelemetry guide). Langfuse documents direct OTel support and version-specific ingestion behavior (official compatibility matrix). Phoenix states that it is built on OpenTelemetry and OpenInference and accepts OTLP traces (official Phoenix overview).
Instrument one representative agent manually before enabling broad auto-instrumentation. That small trace reveals whether the platform can represent loops, parallel tools, fallbacks, and approval waits without flattening the causal chain.
Hosted versus self-hosted is an operating-model decision
Hosted service is usually the fastest route to a shared trace viewer, managed upgrades, and vendor support. Self-hosting can keep telemetry inside infrastructure you control, but it transfers database sizing, backups, upgrades, access control, monitoring, and incident response to your team. “Open source” does not make those operating costs disappear.
The three platforms below span different deployment positions:
| Tool | Hosted option | Self-hosting position | OpenTelemetry position | Evaluation workflow |
|---|---|---|---|---|
| LangSmith | Developer, Plus, and Enterprise plans | Self-hosting is an Enterprise add-on and requires customer-managed infrastructure; the official architecture includes services such as PostgreSQL, Redis, and ClickHouse | Documents OTel ingestion and attribute mapping | Offline and online evaluation, with human, code, LLM-as-judge, and pairwise approaches |
| Langfuse | Hobby, Core, Pro, and Enterprise cloud plans | Core platform is available for self-hosting under the MIT license; some governance features are Enterprise-only | Current SDKs are OTel-based and the platform accepts OTLP | Scores, datasets, experiments, annotation, code evaluators, and LLM-as-judge |
| Arize Phoenix / AX | Phoenix can be self-hosted; Arize AX offers managed tiers | Phoenix is presented as free, open-source, and user-managed; AX Enterprise adds a commercial self-hosted option | Phoenix accepts OTLP and uses OpenInference conventions | Phoenix includes tracing, evaluations, datasets, and experiments; AX adds managed production capabilities |
Sources for these deployment statements: LangSmith self-hosting, Langfuse self-hosted pricing and feature matrix, Phoenix product documentation, and Arize pricing.
Ask for a realistic bill of materials. A self-hosted proof of concept on one container is not evidence that production storage, retention jobs, high availability, authentication, and backups will be simple. Equally, do not reject hosted service solely because prompts leave the application process: first determine whether client-side suppression, regional hosting, or selective tracing can satisfy the actual policy.
Treat prompts and tool payloads as sensitive data
Agent traces can contain more sensitive material than conventional application logs. They may include complete user messages, retrieved document passages, tool arguments, database results, internal URLs, authentication headers, or a model’s generated summary of private material.
Apply privacy controls before export, not after ingestion:
- Classify fields as required, optional, hashed, redacted, or prohibited.
- Disable tracing for workflows that cannot be logged safely.
- Redact in the application or collector before data crosses a trust boundary.
- Keep stable pseudonymous IDs instead of emails, names, or raw account numbers.
- Test nested objects, tool arguments, attachments, metadata, and error messages.
- Set retention by environment and purpose, then test deletion and backup behavior.
- Restrict who can view raw inputs and outputs separately from aggregate metrics.
LangSmith documents client controls that hide or transform inputs, outputs, and metadata, plus conditional tracing for requests that should not be recorded (official sensitive-data guide). Langfuse documents client-side masking and an Enterprise self-hosted ingestion callback; its documentation warns that server-side ingestion masking is not a substitute for client-side masking when data must never leave the application boundary (official masking guide).
For Phoenix, a current first-party page establishing an equivalent built-in redaction control was not confirmed during this research pass: [VERIFY]. Treat collector-side or application-side redaction as a pilot requirement rather than assuming the trace backend will remove sensitive fields.
Retention needs the same scrutiny. Langfuse Cloud currently lists 30 days of data access on Hobby, 90 days on Core, and three years on Pro; configurable retention is listed for Pro, Enterprise, and self-hosted Enterprise (official cloud pricing, official retention documentation). Phoenix 9.0 or later documents time- or count-based policies for self-hosted projects, with indefinite retention as the default (official Phoenix retention guide).
LangSmith has a documentation conflict that must be resolved before publication: its current pricing page describes 14-day base and 180-day extended traces, while another official observability page still states 400-day SaaS retention. [VERIFY] the applicable plan and current retention contract directly from the vendor. Do not build a compliance decision around a comparison article.
Cost attribution must include the observer
An observability platform should show why an agent run cost what it did, not merely display the provider’s total token count. Allocate cost at least by environment, agent version, route, model, customer-safe segment, and tool. Separate four categories:
- model inference used by the agent;
- non-model tools, search, storage, and retrieval;
- evaluator model calls and human review;
- the observability platform and its storage or operations.
Model cost estimates can drift when providers change prices, introduce cached-token tiers, or return incomplete usage. Store the observed token counts, supplied provider cost when available, your price-table version, currency, and calculation method. This lets you recalculate historical estimates instead of overwriting them.
LangSmith says it automatically records token usage and cost for major providers and accepts custom costs for other components (official cost-tracking guide). Langfuse’s observability model includes token and cost tracking, and its Metrics and Observations APIs expose usage and cost fields (official Observations API). Phoenix calculates token-based cost from OpenInference token attributes and model pricing data, with custom price configuration available (official cost-tracking guide).
Published platform prices are only the starting point. On 2026-07-23, LangSmith listed a free one-seat Developer plan with 5,000 base traces per month and a Plus plan at $39 per seat per month with 10,000 base traces, followed by usage charges (official pricing). Langfuse listed Hobby at $0, Core at $29 per month, and Pro at $199 per month, with included units and additional usage conditions (official pricing). Arize’s dedicated page listed AX Free at $0 and AX Pro at $50 per month, with span, ingestion, and retention allowances (official pricing). The broader Arize pricing page showed inconsistent AX Pro allowance and retention values during this check, so those limits remain [VERIFY].
Estimate total monthly cost under normal traffic, a debugging spike, and an evaluation backfill. If high-volume traces are cheap but judge calls are not, a 100% evaluation rate can dominate the bill. Sample routine traffic, preserve all errors and policy violations, and cap evaluation spend separately. For agent-level budget protection beyond the dashboard, see token budgets and circuit breakers.
Compare tools with workload-specific evidence
Do not assign a universal winner. Weight the criteria according to the system you operate:
| Criterion | Pilot question | Evidence to collect |
|---|---|---|
| Trace fidelity | Can it represent loops, parallel tools, retries, handoffs, and sessions? | Same run in application logs and trace UI |
| Instrumentation | Can existing OTel data flow without losing attributes or hierarchy? | Exported sample and field-by-field mapping |
| Evaluation | Can one dataset run code, human, and judge-based checks with versioned scores? | Repeated experiment and regression report |
| Privacy | Can prohibited fields be blocked before export? | Canary secrets and nested-payload redaction tests |
| Retention | Can policy be set, verified, and applied to backups or exports? | Deletion test and documented recovery boundary |
| Cost | Can spend be reconciled by model, tool, route, tenant-safe ID, and evaluator? | Sample invoice and independent calculation |
| Operations | Who handles upgrades, storage growth, authentication, backup, and incident response? | Runbook, ownership, and recovery exercise |
| Portability | Can traces and scores be exported in a usable format? | Successful export and import into a neutral store |
Framework affinity is a tiebreaker, not the whole decision. LangSmith may reduce setup work for teams already centered on LangChain or LangGraph. Langfuse offers an open-source self-hosting path and broad LLM engineering features. Phoenix emphasizes OpenTelemetry/OpenInference, tracing, evaluation, datasets, and experiments. Those are useful starting hypotheses, not substitutes for testing your own trace shape.
If your application routes across multiple providers, include failover runs in the pilot. The decision points in production alternatives to a single model gateway can help define the provider and routing dimensions your observability schema must retain.
Run a seven-day pilot with a fixed scorecard
Use one production-like agent, one representative dataset, and the same instrumentation fields for every candidate. A short controlled pilot is more informative than separate vendor tutorials.
Score each item from 0 to 2: 0 means absent or unusable, 1 means workable with a documented limitation, and 2 means it meets the requirement. Apply weights before seeing the results.
| Pilot area | Weight | Pass condition |
|---|---|---|
| Trace completeness | 20 | At least 95% of sampled runs preserve the expected span hierarchy and terminal status |
| Debugging speed | 15 | An engineer can isolate three seeded failures from trace data without provider-console access |
| Evaluation workflow | 15 | The same versioned dataset produces repeatable code, human, or judge scores |
| Privacy boundary | 20 | No seeded prohibited value reaches the backend; selective no-trace behavior works |
| Cost reconciliation | 10 | Aggregate model cost is within an agreed tolerance of an independent calculation |
| Retention and deletion | 10 | A test project follows the documented deletion policy and known backup boundary |
| Operations and portability | 10 | The team completes an export, restore or redeploy exercise, and writes an owner-backed runbook |
Before the pilot, define the seeded failures: a tool timeout, malformed tool output, repeated retry, retrieval miss, policy stop, and expensive but successful run. Measure trace ingestion delay and dropped spans under load. Record the effort required to add a custom span and a custom evaluator. Run privacy tests with synthetic canary values, never real secrets.
At the end, preserve the raw scorecard, monthly cost model, unresolved [VERIFY] items, and reasons for each weight. Choose the tool that clears your mandatory gates and has the strongest weighted evidence. If no candidate passes the privacy or trace-fidelity gate, improve the instrumentation and repeat the pilot rather than averaging a critical failure into an acceptable total.
Shortlist by the constraint that matters most
The same product can be a strong fit for one operating model and a poor fit for another. Use this table to decide which candidate to test first, not to skip the pilot.
| Primary constraint | Candidate to test first | What must still be proven |
|---|---|---|
| Existing LangChain or LangGraph application | LangSmith | OTel field mapping, data boundary, export workflow, and total usage cost |
| Open-source self-hosting path with a managed-cloud option | Langfuse | Which required features are open-source versus Enterprise, plus database and upgrade ownership |
| OTel/OpenInference-centered instrumentation | Phoenix | Redaction before export, production operations, and whether Phoenix or managed AX matches the workload |
| Strict portability requirement | Test all candidates with the same OTLP trace | Preservation of hierarchy and custom attributes, plus a usable trace-and-score export |
| Strict privacy boundary | Test application-side redaction before selecting a backend | Nested payload masking, no-trace routes, retention, deletion, and backup behavior |
Portability deserves a hands-on test. LangSmith documents trace queries through its SDK or API and recommends Bulk Data Export for large volumes (official trace export documentation). Langfuse documents filtered UI exports in CSV or JSON and separate API and blob-storage paths (official UI export documentation). Phoenix documents downloading spans as a dataframe and querying selected span fields (official span export documentation). These interfaces are not equivalent, so test whether the exported structure is sufficient for replay, audit, and migration rather than checking only for an “Export” button.
Frequently asked questions
Do I need OpenTelemetry to monitor an AI agent?
No. A vendor SDK can be the shortest path to a useful pilot. OpenTelemetry is valuable when you need a shared naming scheme or want to send telemetry through an existing collector, but it does not guarantee identical behavior across backends. Test span hierarchy, custom attributes, links, and exportability with your own trace.
The OpenTelemetry semantic-conventions specification explains that common attribute names make telemetry easier to correlate across codebases and platforms. Its current GenAI attribute registry includes agent, conversation, model, token-usage, retrieval, and tool fields, while marking parts of the GenAI conventions as still evolving (official semantic conventions, official GenAI attribute registry).
Should prompts, outputs, and chain-of-thought be stored in every trace?
Usually not. Store the minimum evidence needed to reproduce failures and evaluate outcomes. Raw prompts, retrieved passages, outputs, and tool payloads may contain personal or confidential data; the OpenTelemetry GenAI registry explicitly warns that input and output message attributes can contain user or PII data. Prefer selective capture, truncation, application-side redaction, stable hashes, and synthetic test values. Do not rely on the observability backend to make unsafe telemetry safe after ingestion.
What sampling rate should an agent observability system use?
There is no universal percentage. Preserve errors, policy stops, approval failures, and rare high-cost routes at a higher rate, then sample repetitive successful traffic according to debugging and evaluation needs. Keep the sampling decision on the trace so dashboards do not mistake a sample for the full population. Recalculate the rate after measuring ingestion volume, incident coverage, and evaluator cost.
Can observability-platform cost be reconciled with the model-provider bill?
It can be compared, but exact equality should not be assumed. Provider invoices, cached-token rules, delayed usage records, missing tool costs, and evaluator calls can create differences. Retain raw usage, currency, the price-table version, and the calculation method; reconcile a small sample of traces against provider records before trusting aggregate dashboards. The LLM API cost calculator guide provides a separate model for estimating inference spend.
What should block a purchase or production rollout?
Treat failed privacy-boundary tests, flattened or missing causal spans, irreconcilable cost data, and an unowned self-hosting runbook as release blockers when those requirements are mandatory. A lower-weight convenience feature should not compensate for a failed hard gate. For evaluation design, start with building an LLM evaluation dataset; for seeded agent failures, the guide to fixing tool-calling errors provides realistic cases to include.
Additional primary sources checked
The following official sources were added during the depth pass and checked on 2026-07-24:
- OpenTelemetry semantic conventions and GenAI attribute registry: https://opentelemetry.io/docs/specs/semconv/ and https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/
- LangSmith trace querying and export: https://docs.langchain.com/langsmith/export-traces
- Langfuse UI export formats and alternatives: https://langfuse.com/docs/api-and-data-platform/features/export-from-ui
- Phoenix span export and query methods: https://arize.com/docs/phoenix/tracing/how-to-tracing/importing-and-exporting-traces/extract-data-from-spans