Kaneki Logo

Ilyass Ezzam

(KanekiEzz)•AI Engineer & Full Stack Developer
← Back to posts
AILLMContext EngineeringAgentsBackendSoftware Architecture

The New Skill in AI Is Not Prompting, It's Context Engineering

21 November 2026·19 min read
Context Engineering

Same Model, Different Results

You have probably lived this story. A demo agent answers every question perfectly. Two weeks into production it ignores instructions, calls the wrong tool, and quotes a document that was updated last month. You rewrite the prompt. It gets a little better, then something else breaks.

The model did not change. What it saw changed. In a real system, the model answers based on a pile of text assembled at that exact moment: instructions, tool descriptions, retrieved documents, memory, and the conversation so far. Most failures come from that pile being wrong, stale, missing, or simply too big.

That is the idea behind context engineering. The term took off in 2025 when people like Shopify's Tobi Lütke and Andrej Karpathy said they preferred it to "prompt engineering", and Anthropic's engineering team now describes it as curating all the tokens that reach the model at inference time: tools, data, and message history included, not just the prompt.

1. Prompt Engineering vs Context Engineering

Prompt engineering asks: how do I word this instruction? Context engineering asks a bigger question: what configuration of information is most likely to make the model behave the way I want, right now?

The context window contains system prompt, tools, retrieved knowledge, memory, conversation history and the user message. Prompt engineering edits only the system prompt box.context windowSystem promptrole, rules, toneToolsfunctions the model can callRetrieved knowledgedocs, files, database rowsMemoryfacts from earlier sessionsConversation historypast turns and tool resultsUser messagethe current requestprompt engineeringedits only this boxLLMreads all of itAnswerContext engineering designs all of these boxes, not just one.
Prompt Engineering Context Engineering
Core question How do I word this instruction? What should the model see right now?
Scope One string Instructions, tools, knowledge, memory, history
When it happens Written once, by hand Assembled at every step, by code
Fails when The wording is ambiguous Information is wrong, stale, missing, or too much
Closer to Writing Systems design

Prompt engineering did not disappear. It became one input of a bigger system, and usually not the one that decides whether your product works.

2. Context Is a Budget, Not a Bucket

Modern models accept huge context windows, so it is tempting to treat them like a bucket: when in doubt, throw it in. In practice this backfires. As the window fills up, the model's ability to recall and use a specific piece of information tends to degrade, a pattern often called context rot. Every extra token draws down a limited attention budget.

The goal is not to fit as much as possible into the window. It is to find the smallest set of high-signal tokens that makes the right behavior likely.

A good rule: every piece of context has to earn its place. If removing it would not make the answer worse, it should not be there.

3. The Ingredients of Context

Ingredient What it is Typical mistake
Instructions System prompt, role, rules, output format Too vague, or too brittle
Tools Functions the model can call, with descriptions Too many, overlapping, unclear names
Knowledge Retrieved documents, files, database rows (RAG) Retrieving 20 chunks "just in case"
Memory Facts stored across sessions Storing everything, never updating
History Previous turns and tool results Letting it grow forever
Task The current request and the expected output Burying it in the middle of everything else

I covered the memory part in my post about chatbot memory architecture: history is what was said, memory is what the system decided to keep. Context engineering is the layer that decides what from all of these actually enters the window at each step.

4. Write Instructions at the Right Altitude

A system prompt can fail in two opposite ways. The first is a brittle prompt: a pile of hardcoded if-else logic written in prose. It breaks on the first case you did not predict.

prompt
You are a support bot. If the user says "refund", ask for the order ID.
If the order ID starts with "A", check table orders_a. If it starts with
"B", check orders_b. If the amount is above 50, reply "escalate". If the
user is angry, say sorry exactly twice. If the user asks about shipping
and the country is not FR, reply with template 7...

The second is a vague prompt that assumes the model shares your context:

prompt
You are a helpful assistant. Help users with their problems and be nice.

The sweet spot is in the middle: specific enough to guide behavior, flexible enough to let the model use its judgment. Organize it in clear sections (XML tags or Markdown headers both work):

prompt
<role>
You are the billing support assistant for Acme. You resolve billing questions.
</role>

<principles>
- Fix the problem in the fewest steps. Ask only for what you cannot look up.
- Refunds under the policy limit are yours to approve. Escalate the rest.
- If you are not sure, say so and show what you checked.
</principles>

<tools>
Always call lookup_order before answering anything about a specific order.
</tools>
  • Start minimal. Test a short prompt first, then add instructions only when you see a real failure.
  • Prefer examples to rule lists. A few diverse, canonical examples communicate expected behavior better than a long list of edge cases.

5. Tools: Fewer, Clearer, Non-Overlapping

Tools are part of the context too: every name, description, and parameter is read by the model on every call. A simple test: if a human engineer cannot say which tool to use in a given situation, the model cannot either.

Problem Better
Five tools that all "search" something Fewer tools with clearly separate jobs and names
Vague parameters like data or input Descriptive names like order_id and start_date
Errors that just say "failed" Errors that say what went wrong and what to try next
Returning a whole database row Returning only the fields the model needs

6. Load Context Just in Time

The classic approach is to preload everything that might be relevant. A better pattern for many agents is just-in-time retrieval: keep lightweight references (file paths, IDs, URLs, saved queries) and let the model load the data with tools only when it needs it.

Preloading everything fills the window with noise and dilutes attention. Just-in-time retrieval keeps light references and fetches only what is needed, giving a small focused context.Preload everythingJust-in-timeDump everything inall docs, all history, all toolsThe window fills uptens of thousands of mixed tokensAttention is dilutedcontext rot: recall dropsSlow, costly, wrongmisses the one key factKeep light referencespaths, IDs, queries, linksModel asks when neededsearch, grep, read fileTool fetches only that2 relevant chunks, not 200Small, focused contextfast, cheap, accurateSame task. The difference is what you let into the window.

In practice the best setups are hybrid: a small, stable core loaded up front (a project instructions file, for example), and everything else fetched on demand. Here is what a context builder looks like when you treat tokens as a budget:

python
def build_context(task: str, user_id: str, history: list, budget: int = 8000):
    facts = memory.search(user_id, query=task, top_k=3)     # long-term memory
    docs = retriever.search(task, top_k=4)                  # only what this task needs

    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},       # short, stable, right altitude
        {"role": "system", "content": "Known facts: " + str(facts)},
        {"role": "system", "content": "Reference material: " + str(docs)},
        *compact(history, keep_last=6),                     # summary + recent turns
        {"role": "user", "content": task},
    ]

    # Every token has to earn its place
    while count_tokens(messages) > budget and docs:
        docs = docs[:-1]                                    # drop the least relevant doc first
        messages[2]["content"] = "Reference material: " + str(docs)

    return messages

Notice that the function is code, not a prompt. That is the point: context is assembled by a program that decides, for each step, what goes in and what stays out.

7. Long Tasks Outgrow the Window

Even with a big window, a long-running agent will eventually produce more tokens than fit: tool outputs, file contents, dead ends. Three techniques keep the context useful over long horizons:

Technique How it works Best for
Compaction Summarize a nearly full context and restart a fresh window with the summary Long back-and-forth conversations
Structured notes The agent writes progress to a file (a NOTES.md or to-do list) outside the window and reloads it later Iterative work with clear milestones
Sub-agents Specialized agents explore in clean windows and return a short summary to a lead agent Parallel research and big explorations

Compaction is only as good as its summary. Tune it for recall first (never lose a key decision), then for precision (cut the fluff):

prompt
Summarize this conversation so work can continue in a fresh context window.

Keep: decisions made, open questions, file paths and names, user preferences,
errors that are still unresolved.
Drop: greetings, repeated tool output, ideas that were tried and rejected.

8. How Context Fails

When an AI system misbehaves, the cause is usually one of these. Learning to recognize them is faster than rewriting the prompt:

Failure Symptom Fix
Missing context Confident wrong answer, hallucination Retrieve it, store it in memory, or let the model ask
Stale context Quotes an outdated policy or document Timestamps, re-fetching, cache invalidation
Noisy context Ignores the relevant fact buried among irrelevant ones Retrieve fewer chunks, rerank, filter
Contradictory context Flips between two different answers One source of truth, resolve conflicts before the prompt
Tool overload Picks the wrong tool, or calls too many Prune, merge, and rename tools
Runaway history Quality drops late in a long session Compaction, summaries, notes

9. A Practical Checklist

  • Print the final context. If you cannot see exactly what the model received at a given step, you cannot debug it. Read it like the model would.
  • Ask of every piece: would the answer get worse if this were removed? If not, remove it.
  • Make the task and critical rules easy to find, not buried in the middle of a long dump.
  • Measure tokens per component (system prompt, tools, documents, history) so you know where the budget goes.
  • Turn real failures into tests. Save the bad conversations and replay them after every change.
  • Change one thing at a time, or you will not know what fixed it.

Conclusion

Prompting is still useful, but it is a small part of the job. The skill that separates a demo from a product is deciding, at every step, what the model should see: the right instructions, the right tools, the right knowledge, the right memory, and nothing else.

So the next time your AI system misbehaves, do not start by rewording the prompt. Print the context and read it. The bug is usually in there.

Further reading: Effective context engineering for AI agents by Anthropic's engineering team.

Chatbot Memory Architecture: How to Remember Message 20 at Message 101

Why trimming and filtering are not enough for production chatbots, and how memory extraction, long-term storage and retrieval let an LLM recall one important fact from a 100-message conversation without re-reading all of it.

Building an AI-Powered Q&A System with LangChain

Step-by-step guide to creating a Retrieval Augmented Generation (RAG) system using FastAPI and LangChain.

Retrieval-Augmented Generation (RAG): Giving LLMs a Memory They Can Trust

A practical introduction to RAG — why LLMs need it, how a basic pipeline is structured, and a working code example for question-answering over your own documents.

← All postsBrowse tags →