A Practical Introduction to RAG: Chat With Your Own Documents

Ask a generic chatbot about your company handbook and it will invent an answer. Ask a retrieval-augmented generation (RAG) system the same question and it can pull the relevant section from the handbook before replying. That is the whole promise: ground the model in documents it never memorised.

Why plain prompts are not enough

Even the largest models have a knowledge cutoff and no access to your private documents. You could paste an entire handbook into the prompt, but context windows have practical limits and cost scales with length. RAG solves both problems: it retrieves only the small slice of text relevant to the question and injects that into the prompt.

The pipeline in four steps

Step 1: Chunk your documents

Split documents into meaningful pieces. A good rule of thumb is 300–800 tokens per chunk, with a small overlap so a sentence is never cut in half. Preserve structure where you can — split on headings and paragraphs rather than arbitrary character counts.

Step 2: Embed the chunks

Each chunk is passed to an embedding model, which returns a vector. Store those vectors in a vector index that supports similarity search. Embeddings map meaning to geometry: chunks about the same topic end up close together.

Step 3: Retrieve on query

When a user asks a question, embed the question with the same model and find the nearest stored vectors. Return the top-k chunks. This is semantic search — it matches by meaning, so "how do I reset my password?" finds the "change credentials" section.

Step 4: Grounded generation

Build a prompt that says: "Answer using only the following context." Append the retrieved chunks and the user's question. Because the model sees the actual source text, it can cite it and it is far less likely to hallucinate.

A minimal implementation

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")

chunks = split_documents(load_handbook())          # step 1
vectors = model.encode(chunks)                     # step 2

def retrieve(query, k=4):
    q = model.encode([query])[0]
    scores = vectors @ q                            # cosine similarity
    top = np.argsort(scores)[::-1][:k]
    return [chunks[i] for i in top]

context = retrieve("How do I reset my password?")
print(build_grounded_prompt(context))               # step 4

The vector math is one line: a matrix product between the query embedding and every chunk embedding. For thousands of chunks that is fine. For millions you need an index like FAISS, pgvector or a managed vector database.

Where RAG breaks down

• Bad chunks: if chunks split mid-context, retrieval returns nonsense. Fix chunking first.
• Wrong embedding model: the retrieval quality ceiling is set by the embedding model. Test a few.
• Keyword-heavy queries: hybrid search (semantic + BM25 keyword) beats pure semantic search for names, IDs and product codes.
• Prompt injection: a retrieved document could contain instructions. Tell the model to ignore instructions inside the context.

Measuring success

Build a set of 50 real questions with the source chunk they should retrieve. Measure retrieval recall (did we find the right chunk?) and then answer correctness (did the model answer well?). Improve retrieval first — no prompt fixes a retrieval miss.

RAG is the difference between an LLM that remembers and one that reads. Most production AI features should start here.