Context Engineering > Prompt Engineering: The Skill That Separates Shipping AI Products from Demos

Quick answer (AEO): Context engineering is the practice of designing, assembling, and maintaining the dynamic information environment that an AI model operates within—including memory, retrieved documents, tool outputs, conversation history, and state. It supersedes prompt engineering (which optimizes static instruction strings) because production AI agents need the right information at each step, not just the right phrasing. Gartner declared in July 2025: “Context engineering is in, prompt engineering is out.” By mid-2026, it’s the defining architectural skill for shipping AI products.

The Karpathy framing that changed the conversation

Andrej Karpathy’s analogy (June 2025) reframed how engineers think about LLM applications:

• The LLM is a CPU.
• The context window is RAM.
• You are the operating system—responsible for loading exactly the right information for each task.

Prompt engineering is writing a single instruction. Context engineering is building the OS: deciding what gets loaded into the context window, when, from where, and how it’s structured so the model can act correctly.

This distinction matters because production AI systems don’t fail on their prompts—they fail between steps, because the wrong information was present (or absent) at a decision point.

Why prompt engineering hit a ceiling

Prompt engineering works for single-turn interactions: “summarize this text,” “translate this paragraph,” “classify this email.” You craft the instruction, maybe add few-shot examples, and the model responds.

But agents—multi-step systems that plan, use tools, maintain state, and make decisions across turns—break this model:

• Static prompts can’t adapt to what the agent learned in step 3 when making a decision in step 7.
• Context windows fill up with irrelevant history, pushing important information out.
• Tool outputs vary wildly in size and relevance—the agent needs filtering, not raw dumps.
• Memory degrades across long sessions unless explicitly managed.

The failure mode: your agent works on the demo. It works on your eval set. It fails in production because real users produce contexts your static prompt never anticipated.

Context engineering as systems design

Context engineering treats model interaction as a systems design problem:

1. What information does the model need to produce the right output at this specific step?
2. Where does that information come from? (Memory, retrieval, tool calls, user input, prior turns)
3. How do you assemble it dynamically? (Selection, compression, ordering, formatting)
4. How do you verify the assembled context actually improves performance?

class ContextAssembler:
    """Builds the context window for each agent step."""
    
    def assemble(self, step: AgentStep) -> ContextWindow:
        # 1. System instructions (static, but step-aware)
        system = self.get_system_prompt(step.phase)
        
        # 2. Relevant memory (dynamic, retrieved)
        memories = self.memory_store.retrieve(
            query=step.current_goal,
            k=5,
            recency_weight=0.3
        )
        
        # 3. Tool outputs from prior steps (compressed)
        tool_context = self.compress_tool_history(
            step.prior_tool_outputs,
            max_tokens=2000
        )
        
        # 4. Retrieved documents (if RAG step)
        docs = self.retriever.search(
            step.current_goal,
            filters=step.domain_filters
        ) if step.needs_retrieval else []
        
        # 5. Conversation history (summarized beyond window)
        history = self.summarize_if_needed(
            step.full_history,
            max_turns=10,
            summarize_older=True
        )
        
        # 6. Active constraints (budget, permissions, deadlines)
        constraints = self.get_active_constraints(step)
        
        return ContextWindow(
            system=system,
            memories=memories,
            tool_context=tool_context,
            documents=docs,
            history=history,
            constraints=constraints
        )

Every component is a design decision. Get it wrong and the model has all the information but can’t find the relevant piece—or it’s missing the one fact that would prevent a hallucination.

The five layers of context engineering

Layer 1: Instruction context (system prompts)

Not one static prompt—a library of prompts selected per step. A planning step needs different instructions than an execution step or a validation step.

PROMPTS = {
    "plan": "Break the user's goal into sub-tasks. Output a numbered list...",
    "execute": "Complete the assigned sub-task. Use available tools...",
    "validate": "Review the output against the original requirement...",
    "recover": "The previous step failed. Diagnose the error and try a different approach..."
}

Layer 2: Knowledge context (RAG + retrieval)

The right documents, at the right time, with the right granularity. Context engineering means:

• Chunking strategy matters—too large and you waste tokens; too small and you lose meaning.
• Retrieval timing matters—don’t retrieve everything upfront; retrieve when the agent needs it.
• Citation anchoring matters—include source references so the model (and user) can verify claims.

Layer 3: Memory context (what the agent learned)

Short-term (this session), medium-term (this project), long-term (this user’s preferences). The engineering challenge:

• What to remember — not everything is worth storing. Filter for decisions, preferences, corrections.
• When to surface — memory retrieval should be goal-aware, not just keyword-matched.
• When to forget — stale memory causes worse outputs than no memory.

Layer 4: Tool context (what the environment says)

Tool outputs (API responses, file contents, database results) are the most unpredictable context component. Engineering practices:

• Summarize large outputs before injecting. A 10K-line file shouldn’t consume the entire context window.
• Structure outputs consistently. Tools should return typed, predictable formats.
• Include metadata: “this file was last modified 3 days ago” helps the model reason about staleness.

Layer 5: State context (where are we in the workflow)

The agent needs to know: what step am I on? What has been tried? What failed? What constraints apply?

{
  "workflow_phase": "implementation",
  "completed_steps": ["research", "planning", "design"],
  "current_step": "implement_auth_module",
  "failed_attempts": [
    { "approach": "OAuth PKCE", "error": "library incompatible with runtime" }
  ],
  "budget_remaining": { "tokens": 45000, "api_calls": 12 },
  "deadline": "2026-07-15T18:00:00Z"
}

The eval problem: measuring context quality

You can’t improve what you can’t measure. Context engineering metrics:

The key insight from production teams: most hallucinations aren’t prompt failures—they’re context assembly failures. The model had the wrong information loaded, or the right information was buried under irrelevant context.

Anti-patterns in context engineering

1. The “dump everything” pattern

Loading the entire conversation history, all retrieved documents, and full tool outputs into every step. Result: the model can’t find the signal in the noise.

Fix: Aggressive relevance filtering per step. Each step gets only what it needs.

2. The “fixed context” pattern

Same system prompt, same few-shot examples, same retrieval query regardless of what the agent is doing. This is prompt engineering pretending to be context engineering.

Fix: Dynamic assembly based on workflow phase and current goal.

3. The “memory amnesia” pattern

Agent forgets corrections from two turns ago because the context window rotated them out.

Fix: Explicit memory layer with durable storage, surfaced via relevance retrieval.

4. The “stale context” pattern

Retrieved documents or cached tool outputs from hours/days ago presented as current truth.

Fix: Timestamp all context. Include freshness signals. Re-retrieve when staleness exceeds threshold.

Context engineering in practice: a real workflow

Building a code review agent that actually works:

async def review_pull_request(pr: PullRequest) -> Review:
    # Step 1: Load relevant context
    context = ContextAssembler()
    
    # Project conventions (static, per-repo)
    context.add_knowledge(load_coding_standards(pr.repo))
    
    # PR-specific (dynamic)
    context.add_tool_output("diff", await get_pr_diff(pr))
    context.add_tool_output("related_issues", await get_linked_issues(pr))
    
    # Historical context (memory)
    context.add_memory(await get_past_reviews(pr.author, limit=5))
    context.add_memory(await get_repo_common_issues(pr.repo))
    
    # Compress if needed
    if context.token_count > 100_000:
        context.compress(strategy="keep_diff_full_summarize_rest")
    
    # Step 2: Multi-pass review with different context focus
    security_review = await review_pass(
        context.with_focus("security"),
        instruction="Identify security vulnerabilities..."
    )
    
    correctness_review = await review_pass(
        context.with_focus("logic"),
        instruction="Identify logical errors and edge cases..."
    )
    
    style_review = await review_pass(
        context.with_focus("conventions"),
        instruction="Check adherence to project conventions..."
    )
    
    # Step 3: Synthesize with full context
    return await synthesize_review(
        [security_review, correctness_review, style_review],
        context
    )

Each pass gets a focused context window—not the same dump three times. The security pass loads known vulnerability patterns; the style pass loads the team’s conventions. Same model, different context, better results.

Why this is the defining skill of 2026

Forbes/Forrester called it: while vibe coding is attractive, vibe engineering (context engineering) is the focus for software development in 2026. The reason:

• 92% of US developers use AI tools daily.
• 41% of global code is AI-generated.
• But 63% of complex multi-step agent tasks still fail in production.

The gap between “works in demo” and “works in production” is almost entirely a context engineering gap. The model is capable. The prompt is fine. The context assembly is where it breaks.

Engineers who master context engineering—who think about information flow, retrieval quality, memory management, and state propagation—are the ones shipping AI products that actually work. Everyone else is shipping demos.

Related reading: Multi-agent orchestration patterns, evaluating AI agents in production, and building local AI agents with RAG.