LLM Observability and Evaluation in Production: Metrics, Tracing, and Guardrails

The 4 Pillars of LLM Telemetry in Production

Traditional APM answers "Did the server return 200 OK?". LLM observability answers"Did the model provide a factual, compliant, cost-effective answer that actually helped the user?"

Instrumenting Distributed LLM Tracing with OpenTelemetry

The OpenTelemetry GenAI Semantic Conventions standardize how prompt tokens, model parameters, and system outputs are captured across microservices. Below is an enterprise-grade TypeScript wrapper for invoking foundational models with native tracing, tenant scoping, and cost attribution:

import { trace, context, SpanStatusCode } from "@opentelemetry/api";
import { countTokens } from "@anthropic-ai/tokenizer";

const tracer = trace.getTracer("ai-agent-service", "2026.1.0");

export async function runTrackedLlmCall({
  userId,
  tenantId,
  prompt,
  systemPrompt,
  model = "claude-3-5-sonnet",
  maxTokens = 2048,
}) {
  return tracer.startActiveSpan("llm.generation", async (span) => {
    const startTime = performance.now();
    const promptTokens = countTokens(systemPrompt + "\n" + prompt);

    span.setAttributes({
      "gen_ai.system": "anthropic",
      "gen_ai.request.model": model,
      "gen_ai.request.max_tokens": maxTokens,
      "gen_ai.usage.prompt_tokens": promptTokens,
      "app.tenant_id": tenantId,
      "app.user_id": userId,
    });

    try {
      const response = await fetch("https://api.anthropic.com/v1/messages", {
        method: "POST",
        headers: {
          "x-api-key": process.env.ANTHROPIC_API_KEY,
          "anthropic-version": "2023-06-01",
          "content-type": "application/json",
        },
        body: JSON.stringify({
          model,
          max_tokens: maxTokens,
          system: systemPrompt,
          messages: [{ role: "user", content: prompt }],
        }),
      });

      if (!response.ok) {
        throw new Error(`LLM provider returned status ${response.status}`);
      }

      const data = await response.json();
      const outputText = data.content?.[0]?.text ?? "";
      const completionTokens = data.usage?.output_tokens ?? countTokens(outputText);
      const totalTokens = promptTokens + completionTokens;
      const durationMs = performance.now() - startTime;

      // Estimate Cost in micro-cents
      const costMicroCents = calculateCost(model, promptTokens, completionTokens);

      span.setAttributes({
        "gen_ai.usage.completion_tokens": completionTokens,
        "gen_ai.usage.total_tokens": totalTokens,
        "gen_ai.response.model": data.model,
        "gen_ai.response.finish_reasons": [data.stop_reason],
        "app.cost_microcents": costMicroCents,
        "app.duration_ms": durationMs,
      });

      span.setStatus({ code: SpanStatusCode.OK });
      return { outputText, promptTokens, completionTokens, costMicroCents, durationMs };
    } catch (error) {
      span.recordException(error);
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: error instanceof Error ? error.message : "Unknown LLM error",
      });
      throw error;
    } finally {
      span.end();
    }
  });
}

By tagging spans with app.tenant_id and gen_ai.usage.total_tokens, engineering teams can immediately build Grafana dashboards or pipe telemetry directly into specialized LLM backends like Langfuse, Arize Phoenix, or Datadog LLM Observability.

Granular Multi-Tenant Cost Attribution

Without strict token budgets, a few rogue enterprise customers or infinite agentic loops can bankrupt an AI SaaS startup in hours. Implement rate-limiting at both the request rate and token budget levels:

• Token Bucket per Tenant: Enforce daily and monthly soft/hard token ceilings in Redis.
• Prompt Compression: Strip redundant whitespace, markdown noise, and repetitive conversation histories before calling LLM APIs.
• Semantic Caching: Cache high-frequency embedding queries in Redis to bypass LLM inference entirely for repeated questions (up to 40% cost reduction).
• Model Cascading: Route easy queries to fast/cheap models (Claude 3.5 Haiku or GPT-4o Mini) and escalate complex queries to frontier reasoning models.

Real-Time Safety Guardrails: Pre-Inference & Post-Inference

Guardrails must run as low-latency checkpoints around the model invocation:

Never allow raw LLM output into user-facing webhooks or database writes without validating structural conformity (e.g., via Zod schemas) and redacting personally identifiable information (credit cards, emails, phone numbers).

Automated Evaluation: Continuous Testing with Ragas & DeepEval

Continuous evaluation compares your AI agent's generated responses against ground truth benchmarks and retrieved contexts. Here is an implementation of an automated asynchronous evaluation step:

import { evaluate } from "ragas-client";
import { detectPII, checkToxicity } from "@guardrails/safety";

export async function runProductionEvaluation({
  question,
  retrievedContexts,
  generatedAnswer,
  groundTruth = null,
}) {
  // 1. Guardrails Layer (Low Latency Pre/Post checks)
  const piiCheck = detectPII(generatedAnswer);
  if (piiCheck.hasSensitiveData) {
    return {
      passed: false,
      reason: "PII_LEAK_DETECTED",
      sanitizedAnswer: piiCheck.redactedText,
    };
  }

  // 2. Automated RAG Evals Layer (Async telemetry pipeline)
  const evalResults = await evaluate({
    metrics: ["faithfulness", "answer_relevance", "context_precision"],
    data: {
      question,
      contexts: retrievedContexts,
      answer: generatedAnswer,
      ground_truth: groundTruth,
    },
  });

  const { faithfulness, answer_relevance, context_precision } = evalResults;

  // Threshold compliance check
  const isHealthy =
    faithfulness >= 0.85 &&
    answer_relevance >= 0.80 &&
    context_precision >= 0.75;

  return {
    passed: isHealthy,
    scores: {
      faithfulness,
      answerRelevance: answer_relevance,
      contextPrecision: context_precision,
    },
  };
}

Detecting Prompt & Model Drift

Foundation model providers regularly push silent updates that change instruction following behaviors, response formats, and safety boundaries. To catch regressions before your customers do:

1. Golden Benchmark Dataset: Maintain a curated dataset of 200–500 realistic prompts covering edge cases.
2. CI/CD Eval Action: Trigger automated evaluation runs on every pull request that touches system prompts, vector chunking, or model configurations.
3. Production Shadowing: Route 5% of production traffic to newly proposed prompts or model versions in shadow mode to compare output quality in real time.

Production Readiness Checklist

Frequently Asked Questions

What is the difference between LLM observability and traditional APM?

Traditional APM tracks server health, latency, and error codes. LLM observability tracks the quality, correctness, token economics, and safety of generative outputs (hallucinations, prompt injections, context relevance).

How can I reduce LLM observability latency overhead?

Use non-blocking asynchronous OpenTelemetry batch exporters (such as OTLP over gRPC) and perform heavy LLM-as-a-judge quality assessments out-of-band using background job workers (e.g., BullMQ or Temporal).

Which metrics are most critical for RAG applications?

The RAG triad: Context Relevance (did the retriever find the right information?),Faithfulness (is the answer grounded strictly in the context?), andAnswer Relevance (did the response directly solve the user query?).