Retrieval-Augmented Generation (RAG): A Practical Guide

A RAG pipeline has two stages. First, retrieval: given a query, search an external index, usually a vector database, sometimes a keyword or graph index too, and return the most relevant chunks of text. Second, generation: pass those chunks to the model along with the original query, so it composes an answer grounded in what was retrieved rather than only what it memorized during training.

The retrieval step is what makes the whole approach work, and it's also where most of the engineering effort goes. Documents get split into chunks, each chunk gets embedded as a vector, and at query time the system finds the chunks whose embeddings are closest to the query's embedding. Get the chunking or the embedding model wrong, and generation quality suffers no matter how good the underlying language model is.

Fine-tuning updates a model's weights on a custom dataset, which is useful for teaching a model a style or a narrow skill, but it's a poor fit for keeping facts current. Every time the underlying facts change, you'd need to retrain, and even then the model can't tell you which fact came from where, since fine-tuning bakes information into weights with no record of provenance.

Retrieval-augmented generation sidesteps both problems. Updating a fact means updating the index, not retraining anything, and the retrieved passages are visible, so you can see exactly what the model based its answer on. That visibility also makes RAG systems easier to audit when something goes wrong, since you can inspect the retrieved context instead of guessing what the model "knows."

Retrieval-augmented generation is only as good as its retrieval step, and pure vector similarity search has real limits. It's good at finding text that's semantically close to a query, but it struggles with multi-hop questions that depend on following explicit relationships between facts, and it can miss exact terms, IDs, or numbers that don't embed distinctively.

RAG also doesn't fully solve hallucination on its own. A model can still misread or over-generalize from retrieved passages, especially if too many marginally relevant chunks get included and dilute the ones that actually matter. That's why retrieval quality, ranking the right chunks highly and excluding the rest, matters more than retrieval volume.

Retrieval doesn't remove the constraint of a context window, it just changes what fills it. Every chunk a retriever selects still has to fit inside the model's context window alongside the query, any system instructions, and the conversation so far, so retrieval is really a budgeting problem: how many chunks, at what length, can be included before the response gets slower and more expensive without adding proportionally more useful information.

This is why raw chunk count is a poor way to measure a RAG pipeline. Returning twenty marginally relevant chunks to fill a large context window usually performs worse than returning five highly relevant ones, since irrelevant text competes for the model's attention and can crowd out the passages that actually answer the question. As Wikipedia's overview of the technique notes, the retrieval step exists specifically to narrow a large corpus down to what's relevant, not to reproduce the whole corpus inside the prompt.

Most RAG problems trace back to retrieval, not generation, so evaluation should start there. Build a small set of representative queries with known-correct source passages, run them through the retriever alone, and check whether the right passage shows up in the top results before ever looking at what the model generates from it. If retrieval misses the right passage, no amount of prompt tuning on the generation side will fix the answer.

Once retrieval looks solid, evaluate the full pipeline on the same queries and specifically check for two failure modes: answers that cite a passage but misstate what it says, and answers that blend two retrieved passages into a claim neither one actually supports. Both are subtler than an outright wrong answer and easy to miss in a quick manual review, which is why retrieval-augmented generation systems headed to production deserve a real evaluation set, not just a handful of spot checks.

Track that evaluation set over time, not just once before launch. A retrieval-augmented generation pipeline that scores well on day one can degrade quietly as the underlying corpus grows, new document types get added, or an embedding model gets swapped without re-indexing everything behind it. Re-running the same evaluation queries after any change to the index, the chunking strategy, or the embedding model is the cheapest way to catch a regression before users do.

Classic RAG implementations lean almost entirely on vector search. A more resilient pattern, often called hybrid retrieval, combines vector similarity with keyword search and graph traversal in a single query, so exact terms and explicit relationships are covered alongside semantic matches, rather than relying on one retrieval strategy to catch everything.

This matters most for agent memory specifically, where a query might need an exact fact ("what's the API key rotation policy"), a semantic match ("anything about security incidents"), and a relationship ("which services does that policy apply to") all at once. Pure vector RAG handles the middle case well and struggles with the other two.

stored's memory graph is queried with hybrid retrieval by default: every search spans graph links, vector similarity, and full-text search, merges the candidates into one ranked set, and, on paid plans, runs a premium reranker over the merged results so the most relevant memories surface first. That's the retrieval half of RAG built in, rather than something you assemble yourself.

Because every memory carries its source and timestamp, you can also see exactly what an agent retrieved before it answered, closing the audit gap that pure RAG pipelines often leave open. Connect an agent over the remote MCP endpoint and retrieval, ranking, and provenance are all handled without standing up a separate vector store.