Large language models are impressive generalists, but ask one a specific question about your company's internal docs, last week's news, or a PDF it's never seen, and it runs into trouble. It might confidently make something up (a "hallucination"), rely on stale training data, or simply have no idea what you're talking about. Retrieval-Augmented Generation — RAG — exists to fix exactly that gap.

Why LLMs Need Help

An LLM's knowledge is frozen at training time and baked into its weights — it can't look anything up, and it has no built-in way to cite where an answer came from. That's a real problem for use cases like Q&A chatbots over private documents, where the model needs to reason about information it was never trained on. RAG solves this not by retraining the model, but by feeding it the right information at the moment it needs it.

How a Basic RAG Pipeline Works

A RAG system has two distinct phases: getting your data ready to be searched (indexing), and actually answering a question using it (retrieval + generation).

Phase 1: Indexing Your Data

1. Load — pull in the raw documents (PDFs, web pages, database rows, whatever the source is).
2. Split — break large documents into smaller chunks (often a few hundred characters each), since retrieval works better on focused pieces of text than on entire documents.
3. Embed — run each chunk through an embedding model, turning it into a vector that captures its meaning numerically.
4. Store — save those vectors in a vector database, built specifically for fast similarity search over high-dimensional vectors.

Phase 2: Retrieval and Generation

This phase runs every time a user asks a question:

1. Embed the question — the user's query gets converted into a vector using the same embedding model from the indexing phase, so it lives in the same "space" as the stored chunks.
2. Retrieve — the query vector is compared against every chunk vector in the database, and the closest matches (by cosine similarity or another distance metric) are pulled out as the most likely relevant context.
3. Generate — the retrieved chunks and the original question are handed to the LLM together, and it generates an answer grounded in that specific context instead of guessing from memory.

A Working Example: Q&A Over a PDF

Here's a minimal RAG pipeline using LangChain, OpenAI embeddings, and a local Chroma vector store — enough to ask questions about any PDF you point it at.

import os
from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.prompts import ChatPromptTemplate
from langchain.chat_models import ChatOpenAI

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
DOC_PATH = "./company_handbook.pdf"
CHROMA_PATH = "handbook_index"

# --- Indexing ---
loader = PyPDFLoader(DOC_PATH)
pages = loader.load()

splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(pages)

embeddings = OpenAIEmbeddings(openai_api_key=OPENAI_API_KEY)
vector_db = Chroma.from_documents(chunks, embeddings, persist_directory=CHROMA_PATH)

# --- Retrieval + Generation ---
question = "How many vacation days do new employees get?"

# pull back the 4 most relevant chunks for this question
results = vector_db.similarity_search_with_score(question, k=4)
context = "\n\n".join(doc.page_content for doc, _ in results)

prompt_template = ChatPromptTemplate.from_template("""
Answer the question using only the context below. If the answer
isn't in the context, say you don't know — don't guess.

Context:
{context}

Question: {question}
""")

prompt = prompt_template.format(context=context, question=question)

llm = ChatOpenAI()
answer = llm.predict(prompt)
print(answer)

Notice the prompt explicitly tells the model not to guess when the context doesn't contain the answer — that instruction matters. Without it, the model will often fall back on its own training data or invent something plausible-sounding, which defeats the point of grounding it in your documents in the first place.

Where This Gets Harder in Practice

A basic pipeline like this works well as a starting point, but a handful of things tend to matter a lot once you move past a toy example:

• Chunk size and overlap — too small and you lose context; too large and irrelevant text dilutes the match.
• Number of retrieved chunks (k) — too few and you might miss the answer; too many and the model gets a noisy, bloated prompt.
• Retrieval quality — cosine similarity on embeddings is a good default, but re-ranking retrieved chunks with a second, more precise model often improves accuracy noticeably.
• Stale indexes — if the underlying documents change, the vector store needs to be updated too, or the model will confidently answer with outdated information.

Conclusion

RAG doesn't make a model smarter — it makes it better informed. By retrieving the most relevant pieces of your own data and handing them to the LLM alongside the question, you get answers grounded in real, current, and specific information instead of whatever the model happened to memorize during training. It's one of the most practical patterns for building LLM applications that actually need to be right, not just fluent.