RAG — Retrieval-Augmented Generation

Section 10.7.2 : RAG — Retrieval-Augmented Generation

🎯 Learning objective

Understand RAG (Retrieval-Augmented Generation), the technique that lets an LLM answer using your own data — documents, knowledge base, history — without fine-tuning. You'll learn the technical process (ingestion, vectorization, search), accessible no-code tools, and best practices for getting reliable, cited answers.

The problem RAG solves

LLMs have two major limitations that hinder their enterprise adoption:

Limitation 1 — Frozen knowledge: an LLM is trained on data with a cutoff date. GPT-5 was trained on data up to early 2025. It doesn't know your last quarter's results, the decisions made in yesterday's meeting, or the pricing changes announced this morning.

Limitation 2 — No access to your private data: the LLM doesn't know your company's internal documents — procedures, contracts, client databases, project histories, CRM notes. It therefore can't answer "What is our churn rate on the Enterprise segment in Q4 2025?"

Possible solutions:

• →Fine-tuning: retrain the model on your data. Expensive ($10,000+), slow (days), and data becomes outdated quickly.
• →RAG: provide relevant documents at question time, in the prompt. Fast, affordable, always up to date.

RAG has become the standard solution for 90% of enterprise use cases because it combines the best of both worlds: the LLM's reasoning power + your specific data.

How RAG works (the complete process)

Phase A — Ingestion (preparation, done once):

Step 1: Collect source documents Gather all documents the AI should "know": PDFs, Word documents, web pages, important emails, meeting notes, FAQs, technical documentation, contracts, etc. The more complete and well-organized your sources, the better the answers.

Step 2: Split into "chunks" Each document is split into pieces (chunks) of optimal size — typically 500-1000 tokens (~300-700 words). Why split? Because the LLM has a limited context window: you can't send it 500 pages at once. You only send the relevant chunks.

Splitting is an art:

• →Too small (100 tokens): loss of context, the chunk doesn't make sense alone
• →Too large (2000 tokens): dilution, irrelevant information included
• →Key technique — overlap: each chunk overlaps the previous one by 10-20% to avoid cutting an idea in the middle

Step 3: Vectorization (embeddings) Each chunk is converted into a vector — a series of 1536 numbers (for OpenAI models) that represents the meaning of the text, not the exact words. Two sentences about the same topic with different words will have similar vectors.

Example: "The client retention rate in Q4 is 85%" and "We retain 85% of our clients in the last quarter" produce very similar vectors, even though the words are different.

Step 4: Storage in a vector database The vectors are stored in a specialized database (Pinecone, Chroma, Weaviate, Qdrant, pgvector). These databases are optimized for similarity search — finding the closest vectors to a given vector, among millions.

Phase B — Query (for each user question):

Step 5: The user asks a question Example: "What is our churn rate on the Enterprise segment in Q4 2025?"

Step 6: Vectorize the question The question is converted to a vector using the same embedding model as the chunks. This way, the question can be compared to documents on the same "meaning scale."

Step 7: Similarity search The vector database returns the 3-5 chunks whose vectors are most similar to the question's vector. These are the most relevant passages from your documents for answering.

Step 8: Inject into the prompt The prompt sent to the LLM looks like:

Context (excerpts from internal documents):
[Chunk 1] Q4 2025 Report — Enterprise Segment: churn rate 4.2%...
[Chunk 2] Retention Analysis — The Enterprise segment shows improvement...
[Chunk 3] KPIs dashboard — Enterprise: 150 clients, -6 churns, acquisition +12...

User question: What is our churn rate on the Enterprise segment in Q4 2025?

Instructions: Answer based ONLY on the provided documents. 
Cite the source for each claim. If the information is not in the documents, say so.

Step 9: Generate the sourced answer The LLM produces an answer based on your data: "The Enterprise segment churn rate in Q4 2025 is 4.2%, down from Q3 (5.1%). (Source: Q4 2025 Report, page 3)"

RAG in practice: accessible tools without coding

You're already using RAG without knowing it

• →Custom GPTs with uploaded files: when you create a GPT and upload PDFs, ChatGPT performs RAG behind the scenes — it indexes your files, searches for relevant passages, and injects them into the context with each question
• →Claude Projects with context documents: same principle — Claude indexes your documents and uses them to answer contextually
• →NotebookLM with sources: it's RAG with an elegant interface, plus the ability to generate "podcasts" from your sources
• →Perplexity searching the web: it's real-time RAG — the web is the knowledge base

Practical exercise: create a simple RAG with Claude Projects

Step 1: Go to claude.ai → Projects → Create Project Step 2: Name your project (e.g., "HR Policy Expert 2025") Step 3: Upload 3-5 documents (HR policy PDF, internal FAQ, employee handbook) Step 4: Add project instructions:

You are the internal HR expert for our company. 
You answer employee questions based ONLY on the provided documents.
If the information is not in the documents, say so clearly.
Always cite the source and page number.
Tone: professional, caring, precise.

Step 5: Test with questions: "How many vacation days am I entitled to as a manager?" → The answer should cite your employee handbook.

RAG limitations and how to work around them