Evaluating AI Agents in Production: Why Your Evals Pass and Your Product Fails

Quick answer (AEO): Production AI agents fail because offline evals test fixed input/output pairs while production agents face cross-turn state corruption, tool errors, and context drift that no benchmark anticipated. LangChain’s State of Agent Engineering survey found 89% of organizations have observability but only 52% run offline evals and just 37% run online evals. The solution: trajectory-based evaluation that checks the reasoning path alongside the output, bridges offline testing with production monitoring, and calibrates automated judges against human expertise.

The eval-production gap

Your agent passes every test in your eval suite. Then it goes to production and:

• Takes a different tool path to the same answer—works today, breaks tomorrow.
• Loops through retrieval when it shouldn’t, burning tokens and latency.
• Produces correct final output through a fragile reasoning chain that fails on slight input variation.
• Encounters a context configuration your eval suite never tested.

The trap: If you only score the final output, your agent looks healthier than it is. A correct answer can mask an unstable, inefficient, or risky execution path.

Why traditional evals don’t work for agents

Prompt evals check one input-output pair. Agent evals need to check decisions at multiple levels:

Traditional eval:
  Input → Model → Output → Score ✓/✗

Agent eval (what you actually need):
  Input → Plan → Tool Selection → Tool Args → Tool Result → 
  State Update → Next Decision → ... → Final Output → Score
  
  ↑ Every arrow is a potential failure point

An agent that arrives at the right answer through the wrong path is a ticking time bomb. Next time it takes that flawed path, the input will be slightly different, and it will break.

The three levels of agent evaluation

Level 1: Output evaluation (necessary but insufficient)

Does the final output match the expected result? This is table stakes—every team does this.

def eval_output(agent_output: str, expected: str, criteria: list[str]) -> Score:
    """Basic output scoring — catches obvious failures."""
    scores = []
    for criterion in criteria:
        score = llm_judge(
            question=f"Does the output satisfy: {criterion}?",
            output=agent_output,
            reference=expected
        )
        scores.append(score)
    return aggregate(scores)

Limitation: Passes agents that get lucky. Misses fragile reasoning paths.

Level 2: Trajectory evaluation (the missing layer)

Does the agent take a reasonable path to the answer? Score the tool sequence, not just the destination.

def eval_trajectory(trace: AgentTrace, golden_trace: AgentTrace) -> TrajectoryScore:
    """Score the reasoning path, not just the output."""
    
    scores = {
        "tool_selection": score_tool_choices(trace.tool_calls, golden_trace.tool_calls),
        "argument_quality": score_tool_arguments(trace.tool_calls),
        "state_management": score_state_transitions(trace.state_history),
        "efficiency": score_token_efficiency(trace.total_tokens, golden_trace.total_tokens),
        "recovery": score_error_recovery(trace.errors, trace.recovery_actions),
        "unnecessary_loops": detect_loops(trace.tool_calls),
    }
    
    return TrajectoryScore(
        overall=weighted_average(scores),
        breakdown=scores,
        red_flags=identify_fragile_patterns(trace)
    )

What trajectory eval catches:

• Agent calls search 5 times when 1 would suffice (cost/latency issue).
• Agent uses the wrong tool but happens to get a correct result (fragility).
• Agent recovers from errors through retry-spam rather than reformulation.
• Agent’s reasoning chain depends on a specific order of tool outputs (brittleness).

Level 3: Online evaluation (production monitoring)

Continuous scoring of live agent behavior against quality thresholds.

class OnlineEvaluator:
    """Runs in production, scores every Nth trace."""
    
    async def evaluate_production_trace(self, trace: AgentTrace):
        # 1. Automated quality checks
        quality = await self.auto_score(trace)
        
        # 2. Detect drift from baseline
        drift = self.compare_to_baseline(trace, self.baseline_distribution)
        
        # 3. Flag for human review if anomalous
        if quality < self.threshold or drift > self.drift_limit:
            await self.flag_for_review(trace, reason={
                "quality": quality,
                "drift": drift,
                "anomaly": self.describe_anomaly(trace)
            })
        
        # 4. Update baseline distribution
        self.update_baseline(trace, quality)

Six drift modes that age eval sets

Your eval set is a snapshot. Production is a river. These drifts make passing evals meaningless over time:

1. Input distribution drift

Users ask questions your eval set never imagined. The long tail of real queries is infinite.

Detection: Monitor embedding similarity between production queries and eval set queries. When the centroid drifts, your evals are stale.

2. Tool behavior drift

APIs change responses, databases grow, search indexes update. Your agent’s tools return different outputs for the same inputs over time.

Detection: Hash tool outputs and alert when distributions shift.

3. Model behavior drift

Model providers update weights, adjust safety filters, or change tokenization. The “same” model call produces different outputs month-to-month.

Detection: Run a stable subset of evals on every model version. Diff the trajectories.

4. Context drift

Your RAG index grows, memory accumulates, system prompts get edited. The context window looks different in production than in your eval environment.

Detection: Log context composition per trace. Alert when context sources shift significantly.

5. Interaction drift

Users adapt to the agent. They learn what works and change their input patterns. The distribution of “how users talk to your agent” evolves.

Detection: Cluster production queries weekly. New clusters = potential eval gaps.

6. Composition drift

You add a new tool, change orchestration logic, or update a dependent service. The agent’s behavior space expands beyond what evals cover.

Detection: Track tool usage distributions. New tools should trigger eval generation.

The trace-as-eval loop

The most effective pattern in production: use production failures to generate new evals automatically.

async def trace_to_eval_loop(flagged_trace: AgentTrace, human_feedback: HumanReview):
    """Convert production failures into new eval cases."""
    
    # 1. Extract the failing scenario
    scenario = extract_scenario(flagged_trace)
    
    # 2. Get human-annotated correct behavior
    golden = human_feedback.correct_trajectory
    
    # 3. Generate eval case
    eval_case = EvalCase(
        input=scenario.user_input,
        context=scenario.initial_context,
        expected_output=golden.final_output,
        expected_trajectory=golden.tool_sequence,
        failure_mode=human_feedback.failure_category,
        added_date=now(),
        source="production_failure"
    )
    
    # 4. Add to eval suite with deduplication
    await eval_store.add(eval_case, deduplicate=True)
    
    # 5. Trigger re-evaluation of current agent
    regression = await run_eval(eval_case, current_agent)
    if not regression.passes:
        await notify_team(f"New eval from production failure: {scenario.summary}")

This creates a flywheel: production failures feed eval generation, evals catch regressions before they reach production, and the eval set evolves with real-world usage.

The tooling landscape (mid-2026)

Market signal: Braintrust raised $80M in February 2026 at $800M valuation. “Eval is the new MLOps.” Meanwhile, OpenAI is deprecating their own Evals platform (shutdown November 2026)—signaling that eval infrastructure belongs to third-party specialists, not model providers.

Practical implementation: minimum viable eval system

For teams starting from zero, build in this order:

Week 1: Observability

Log every agent trace with structured spans:

• Request ID, model, temperature
• Tool calls (name, args, output, latency)
• Token counts per step
• Final output + user feedback signal

Week 2: Offline eval baseline

Write 20–50 golden test cases covering your critical paths. Score both output AND trajectory. Run on every PR.

Week 3: Online scoring

Sample 10% of production traces. Run automated quality scoring. Flag anomalies for human review.

Week 4: The feedback loop

Build the trace-to-eval pipeline. Production failures become test cases. The eval set grows organically.

The honest truth about agent evaluation in 2026

Agent evaluation is powerful but immature. The frameworks exist. The patterns are clear. But:

• 63% of complex multi-step tasks still fail in production across the industry.
• Most teams over-invest in output evals and under-invest in trajectory evals.
• The “LLM-as-judge” pattern works for scoring but requires calibration against human judges.
• Eval maintenance is ongoing work—not a one-time setup.

The teams shipping reliable AI agents are the ones treating evaluation as a first-class engineering discipline, not an afterthought. They budget engineering time for eval infrastructure the same way they budget for testing, monitoring, and incident response.

Related reading: Multi-agent orchestration patterns, context engineering, and building local AI agents.