Building RAG Systems That Don't Hallucinate

Everyone is building RAG. Most of it is broken. The demo looks great — ask a question, get a relevant answer, ship it. Then a user asks something slightly off-script and the model invents a policy that doesn't exist, cites a document it never retrieved, or confidently states the opposite of what your knowledge base says. This is not an LLM problem. It is an architecture problem.

Naive RAG is three steps: embed the query, retrieve top-k chunks by cosine similarity, stuff them into the prompt. This fails in predictable ways.

First, chunking by fixed token count splits ideas across chunk boundaries. A sentence that begins in chunk 14 and ends in chunk 15 will never be retrieved as a unit. Your retriever finds half an answer and the model fills in the rest — with fabrication.

Second, cosine similarity on raw embeddings rewards lexical overlap, not semantic relevance. A question about "contract termination clauses" may retrieve every chunk containing the word "contract" while missing the one document that actually defines termination conditions.

Third, top-k retrieval has no quality floor. If nothing in your corpus is genuinely relevant, you still get k chunks shoved into the context. The model treats them as authoritative.

The goal of chunking is to produce units that are semantically self-contained. Fixed token windows do not do this.

Sentence-window chunking embeds at the sentence level but retrieves surrounding context. You get precise retrieval with enough context for the model to reason.

Hierarchical chunking creates two tiers: small chunks for retrieval, larger parent chunks for context injection. Match on the small chunk, return the parent. This eliminates split-idea failures.

Document-aware chunking respects structure — headers, list items, code blocks, tables. A markdown parser that understands that a bullet point belongs to its parent heading will always outperform a naive splitter.

Retrieval narrows your corpus to candidates. Reranking finds the right ones.

A cross-encoder reranker takes each (query, chunk) pair and scores it jointly — unlike bi-encoders that score independently. Cross-encoders are slower but dramatically more accurate. Run retrieval to get top-20, rerank to top-5, pass top-5 to the model.

Cohere Rerank, BGE reranker, and Jina Rerank are all production-ready. The latency overhead is 100-200ms. The accuracy gain is worth it every time.

The hardest part of RAG is knowing whether it works. Manual spot-checking finds bugs. It does not find distributions.

Build an evaluation set: 50-100 questions with ground-truth answers and the source documents they should come from. Measure three things: retrieval recall (did the right document appear in retrieved context?), answer faithfulness (is the answer grounded in the retrieved context, not hallucinated?), and answer relevance (does the answer actually address the question?).

RAGAs is a good framework for this. Run it before every deployment. Track it over time. When answer faithfulness drops, you have a retrieval problem — fix the chunking. When retrieval recall drops, you have an embedding problem — retrain or switch models.

Use semantic or hierarchical chunking. Add a reranker between retrieval and generation. Build an eval set before you launch. Measure continuously.

Naive RAG is a demo. Production RAG is an engineering discipline. If you're building an AI product where accuracy matters — and it always matters — treat it like one.

We've built RAG systems for fintech, legal, and healthcare clients where hallucinations have real consequences. If you're working on something similar, we're happy to talk through your architecture.