When a RAG system gives a bad answer, it’s easy to blame the LLM. But the model is rarely the only culprit.
Sometimes the source data is just out of date. Other times, poor chunking slices a fact right down the middle, destroying its meaning. Or your search relies too heavily on exact keywords, missing what the user actually meant. And even if you find the right information, it might get buried under a pile of irrelevant text by the time it reaches the prompt.
Simply put: you can't prompt-engineer your way out of a broken retrieval pipeline.
Building a reliable RAG system means treating it like a complete engineering pipeline, not just an API call to a model. The final answer is only as good as what you ingest, how you chunk it, what you retrieve, how you rank it, and what context actually makes it to the prompt.
Take search, for instance. Dense vector search is great for understanding the overall meaning or "vibe" of a query, but it can fail spectacularly at finding specific part numbers or acronyms. That’s where a sparse retrieval method like BM25 shines. By combining both approaches and using a strategy like Reciprocal Rank Fusion, you get a much stronger, well-rounded context set than you would using either method alone.
We also have to think about how users actually ask questions. A single, quickly typed query rarely captures all the ways that information might be stored in a database. Using techniques like multi-query expansion to generate a few different variations of the user's prompt can drastically increase your chances of pulling the right evidence.
But you can't rely on gut feelings to know if your retrieval is actually getting better.
You have to measure it. Metrics like context recall and context precision tell you exactly if you're pulling enough information, and if that information is actually useful. Tools like RAGAS take the guesswork out of this, turning subjective impressions into repeatable benchmarks.
This is especially true once you're in production. When a system hallucinates or gives a wrong answer, you need to be able to look under the hood and ask: What documents did we index? Which exact chunks got pulled? How did the reranker score them? Did the model even have the right facts to work with?
That’s why observability isn't a nice-to-have; it's a requirement to answer those questions.
Real RAG engineering happens long before the model generates a single word. It starts with cleaning your data, structuring your documents, optimizing your retrieval, and building in solid evaluation and observability.
Once you can actually measure those stages, improving your AI system stops being a game of endlessly tweaking prompts. Instead, it becomes a systematic process of feeding the model better information.
The model is still important, of course. But in any grounded AI system, the quality of your context almost always determines the quality of your answer.