Kaneki Logo

Ilyass Ezzam

(KanekiEzz)•AI Engineer & Full Stack Developer
← Back to posts
AILLMChatbotMemoryLangGraphBackendSoftware Architecture

Chatbot Memory Architecture: How to Remember Message 20 at Message 101

8 November 2026·14 min read
Chatbot Memory Architecture

The Problem: A Chatbot That Forgets

Imagine this conversation with a support chatbot:

Message Content
1 to 19 Small talk and general questions
20 My API URL is https://api.example.com
21 to 100 Many unrelated questions and answers
101 What is my API URL?

An LLM has no memory of its own. It only sees what you send it in each request. Sending all 101 messages works for a demo, but in production it is slow, expensive, and eventually impossible because of the context window limit. So you cut the history. The most common way to do it is also the most dangerous:

python
recent = messages[-3:]
response = llm.invoke(recent)

Now the model only sees the last three messages. Message 20 is gone, and the model either says it does not know or, worse, invents a URL.

Trimming keeps the last three messages and loses message 20Trimming forgets what mattersMessages 1-19small talkMessage 20API URL lives hereMessages 21-98unrelatedLast 3 messagesmessages[-3:]The API URL never reaches the modelLLM sees only thisTrimming decides what to send. It has no idea what is important.

Why Filtering and RemoveMessage Are Not Enough

If you build chatbots with LangGraph or LangChain you quickly meet the standard toolbox: filtering, RemoveMessage, trimming, summarization. They are all useful, but they answer a different question from the one we have.

  • Filtering chooses which messages are sent to the LLM. If the important one is not selected, it is lost.
  • RemoveMessage deletes messages from the graph state. Whatever information they held is deleted with them.
  • Trimming keeps the context under a token budget, but it cuts by position, not by importance.

All of these treat the conversation as history: what was said. What we actually need is memory: what the system must keep and be able to bring back later.

History is what was said in the conversation. Memory is what the system decided to keep and can retrieve in the future. They are not the same thing.

The Solution: Memory Extraction

Instead of hoping that an important fact survives inside the message window, the system extracts it at the moment the user says it and stores it in a structured form. When the user sends message 20:

python
My API URL is https://api.example.com

a memory extraction step produces:

python
{
  "api_url": "https://api.example.com"
}

and saves it in a database or a long-term memory store. From now on you can delete message 20 from the state, and the information is still safe.

Three Ways to Decide What to Store

Approach How it works Tradeoff
Rules Regex or pattern matching, for example pull a URL out of any message that mentions an API URL Fast, cheap and predictable, but only catches cases you thought about
LLM extraction Ask an LLM to read the message and return the important facts as JSON Flexible and understands any phrasing, but costs tokens and must be validated
Hybrid Rules for well-defined data (emails, URLs, IDs), LLM for everything else Best balance for most production systems

Here is a minimal LLM-based extractor. The important details are the strict JSON output, the empty object for messages with nothing to remember, and the upsert so a corrected value replaces the old one:

javascript
import json

EXTRACTION_PROMPT = """
Extract durable facts the user states about themselves, their project
or their preferences. Ignore small talk and temporary information.
Return ONLY a JSON object of key/value pairs, or {} if there is nothing.

Message: {message}
"""

def update_memory(user_id: str, message: str) -> None:
    raw = llm.invoke(EXTRACTION_PROMPT.format(message=message)).content
    try:
        facts = json.loads(raw)
    except json.JSONDecodeError:
        return  # never let a bad extraction break the chat

    for key, value in facts.items():
        memory_store.upsert(user_id, key, value)  # new value replaces old

Memory Retrieval: Getting the Right Fact Back

Storing is only half the job. When message 101 arrives, What is my API URL?, the system uses the question to search the memory and bring back only what is relevant. The LLM never needs to see the other 100 messages.

Remember: message 20 is extracted and stored in long-term memory. Recall: message 101 triggers retrieval, the fact is added to the prompt, and the LLM answers.1231234RememberRecallMessage 20ExtractionStorageMessage 101RetrievalPromptAnswerMy API URL isapi.example.comDetect what mattersRulesLLMHybridthen save as JSON factsLong-term memoryapi_url = api.example.comlanguage = enWhat is myAPI URL?Search memorytop_k = 3, not 100 msgsRecent messagesSummaryMemory: api_urlQuestionYour API URL isapi.example.comOnly the relevant fact reaches the model, not all 100 messages.

Retrieval can be implemented in three ways:

  • Metadata / structured lookup. The memory is a table or key-value store, and you fetch api_url directly. Perfect for well-defined facts.
  • Semantic search with embeddings. You embed the question and find the closest stored memories. Useful when the question does not use the same words as the stored fact.
  • Hybrid retrieval. Use both: structured lookup for known keys, semantic search for the fuzzy rest.
python
def build_prompt(user_id: str, messages: list, summary: str) -> list:
    question = messages[-1].content
    memories = memory_store.search(user_id, query=question, top_k=3)

    memory_block = "\n".join(f"- {m.key}: {m.value}" for m in memories)
    system = (
        "You are a helpful assistant.\n"
        f"Conversation summary: {summary}\n"
        f"Known facts about the user:\n{memory_block}"
    )
    return [{"role": "system", "content": system}, *messages[-3:]]

Message 101 now reaches the model with api_url in its context, even though message 20 was deleted a long time ago.

Putting It All Together

The old techniques still matter. The point is that each one has a clear job, and none of them is a memory system on its own:

Technique Role
Filtering Choose which messages are sent to the LLM
RemoveMessage Delete messages from the graph state
Trimming Keep the context under a token limit
Summarization Compress the old part of the conversation
Long-term memory Store the important facts in a structured form
Retrieval Bring back only what the current question needs

A practical prompt for a production chatbot is assembled from four layers:

  1. Recent messages so the model understands the current dialogue.
  2. Summary of the older conversation.
  3. Long-term memory with important, structured facts.
  4. Retrieved context containing only what relates to the current question.

Production Notes

  • Do not store everything. Temporary or unimportant details turn memory into a second, noisier chat history.
  • Handle sensitive data carefully. Passwords, tokens and personal data need explicit rules: what is stored, where, for how long, and how the user can delete it.
  • Handle updates. When the user corrects a fact, replace the old value instead of keeping two conflicting ones.
  • Evaluate extraction. Test whether the system stores the right facts and does not store wrong ones. A bad memory is worse than no memory.

Conclusion

Trimming and filtering control the size of the context. They do not decide what is worth remembering. A real memory architecture separates two things: the conversation history, which can be cut, summarized and deleted freely, and the memory, which holds the facts the system must keep and can retrieve when needed.

That difference is the step from a simple chatbot to one with a true memory architecture, and it is worth understanding well before you build RAG pipelines or multi-agent systems that run long conversations.

The New Skill in AI Is Not Prompting, It's Context Engineering

Why the quality of an AI system depends less on how you phrase one instruction and more on what the model sees at every step: system prompts at the right altitude, tools, just-in-time retrieval, memory, compaction, sub-agents, and the failure modes of a bad context.

Building an AI-Powered Q&A System with LangChain

Step-by-step guide to creating a Retrieval Augmented Generation (RAG) system using FastAPI and LangChain.

Retrieval-Augmented Generation (RAG): Giving LLMs a Memory They Can Trust

A practical introduction to RAG — why LLMs need it, how a basic pipeline is structured, and a working code example for question-answering over your own documents.

← All postsBrowse tags →