RAG: teach Claude about your world

The 3 problems RAG solves

• Problem 1: Claude doesn't know your company. It was trained on public internet data up to a cutoff date. It never saw your internal policies, contracts, Notion, CRM, or recent data. Ask about those → hallucination.
• Problem 2: Retraining the model is not an option. Training cost: millions of dollars, weeks of compute. And every time a document changes, retrain again. Impossible.
• Problem 3: Putting everything in context doesn't work either. 10,000 docs × 500 tokens = 5 million tokens per request. Not technically or economically feasible. And with that much context, "lost in the middle" hits catastrophically.

The solution: search first, generate second

Without RAG:
  question → Claude generates from its parameters → answer
  (Claude invents something plausible if it doesn't know)

With RAG:
  question → find relevant documents → inject only those into context → Claude generates → answer
  (Claude responds based on real documents)

The 4 internal steps of RAG

• Step 1 — Ingestion (once): chunk documents into ~500 token fragments, convert each chunk into a numeric vector (embedding), store in vector database.
• Step 2 — Retrieval (every query): convert the user question into a vector, find the chunks whose vector is most similar (cosine similarity), return the top N.
• Step 3 — Augmentation: insert the retrieved chunks into the LLM context alongside the original question.
• Step 4 — Generation: Claude generates the response conditioned on real evidence, not statistical parameters.

Chunking: the step most people get wrong

Chunking is splitting documents into fragments before embedding them. If you chunk badly, the retrieval fails no matter how good the rest is.

• Too small (50 tokens): chunks lose context. A sentence without its surrounding paragraph means little.
• Too large (2000 tokens): the semantic signal gets diluted. The chunk covers too many topics, similarity scores become meaningless.
• Good default: 400-600 tokens with 10-20% overlap between consecutive chunks. Overlap ensures a sentence cut in the middle still appears whole in at least one chunk.

RAG vs SQL: when to use each

"Show contracts signed in 2023"
→ SQL. Exact search by value.

"Show contracts with penalty clauses for delays"
→ RAG. Semantic search. SQL can't do this.

"How many active users do we have in Spain?"
→ SQL. Exact number from database.

"What are users saying about checkout usability?"
→ RAG over support tickets or reviews.

Rule: if you can express the search as an exact condition → SQL.
If you're searching by meaning or conceptual similarity → RAG.
In real systems, you use both.

What RAG does NOT solve

RAG has two independent components that can fail separately. Bad retrieval + good generation = wrong answer with confidence. Good retrieval + bad reasoning = wrong answer from good information. If the retrieval fails (because the question uses different terminology than the document, or because chunking broke the relevant context), the generation also fails. RAG solves access to information — not reasoning errors.