Chunking Strategies RAG: A Practical Guide

A vector embedding represents the meaning of whatever text it's computed over, one embedding per chunk. If a chunk contains one coherent idea, its embedding represents that idea well. If a chunk contains three unrelated ideas because it was split at an arbitrary character count, its embedding ends up as a blurry average of all three, which retrieves worse for any single one of them.

This is why chunking strategy shows up disproportionately in postmortems of underperforming RAG systems. Improving the embedding model or the retrieval algorithm rarely helps if the chunks being embedded were poorly formed to begin with, the original RAG paper itself treated the granularity of retrieved passages as a first-order design decision, not an afterthought.

The simplest strategy splits text into chunks of a fixed token or character count, often with some overlap between consecutive chunks so a fact split across a boundary still appears intact in at least one chunk. It's trivial to implement and works reasonably well as a baseline, especially on content without strong internal structure.

Its weakness is that it ignores meaning entirely: a fixed-size cut can land in the middle of a sentence, split a table from its caption, or separate a claim from the qualifier that changes its meaning. For anything beyond a rough baseline, fixed-size chunking is usually the first thing worth improving on.

Semantic chunking splits text at natural meaning boundaries instead of a fixed size, typically by measuring similarity between adjacent sentences and cutting where the topic shifts rather than at an arbitrary character count. The result is chunks that correspond to coherent ideas, which embed and retrieve more precisely than fixed-size chunks that happen to straddle two unrelated topics.

The cost is complexity: semantic chunking requires an extra embedding or similarity-scoring pass over the source text before chunking is even done, and chunk sizes end up variable, which some downstream systems handle less gracefully than a predictable fixed size.

Structure-aware chunking uses the document's own structure, headings, paragraphs, list items, code blocks, as natural chunk boundaries rather than inferring them from content alone. For structured content like documentation or code, this is often the highest-quality option, since the author has already marked the meaningful divisions for you.

It's less useful for unstructured text with no clear formatting, a raw conversation transcript, for example, where semantic or fixed-size chunking is the more practical fallback. Most production pipelines mix strategies: structure-aware where structure exists, semantic or fixed-size elsewhere.

AI agent memory sidesteps a lot of the chunking problem by not chunking documents at all, it stores discrete facts and entities as they're learned, rather than splitting a large document after the fact. Each memory is naturally chunk-sized because it was written as one atomic fact to begin with, which avoids the whole class of problems caused by cutting a document in the wrong place.

That's a meaningfully different approach from document-oriented RAG, where chunking is unavoidable because the source content, PDFs, wikis, long reports, wasn't authored as discrete facts. For memory specifically, getting the write-time granularity right matters more than getting chunking right after the fact.

Beyond picking a chunking strategy, two parameters need tuning regardless of which one you use: chunk size and overlap. Chunk size controls the tradeoff described above directly, larger chunks preserve more context per vector embeddings computation but blur-sm more ideas together, while smaller chunks embed cleaner but risk losing context a reader, or a model, needs to interpret the fact correctly.

Overlap addresses a specific failure mode common to every one of the chunking strategies RAG systems use: a fact or sentence that happens to fall right on a chunk boundary. A modest overlap, repeating the tail of one chunk at the start of the next, means that fact still appears whole in at least one chunk instead of being split in half and effectively lost to retrieval. The right overlap is usually a fraction of chunk size, enough to catch boundary-straddling content without meaningfully inflating storage or retrieval cost.

A useful rule of thumb: start with roughly ten to twenty percent overlap relative to chunk size, then adjust based on how often your evaluation queries hit a boundary-split fact. Too little overlap and boundary splits go uncaught; too much and you're storing and searching over largely duplicated content, paying storage and retrieval cost twice for the same information without a corresponding quality gain.

In practice, most teams don't need to pick one chunking strategy for their entire corpus. The original Retrieval-Augmented Generation framing treated retrieval granularity as something to tune per source, and that still holds: structure-aware chunking for well-formatted documentation, semantic chunking for long-form prose without clear headings, and fixed-size chunking with overlap as the fallback for anything else.

The most reliable way to choose isn't theoretical, it's measuring retrieval quality on real queries against your actual content with each candidate approach, since the chunking strategies RAG teams settle on for one kind of document can perform noticeably worse on another. A small evaluation set of representative queries with known-correct answers is usually enough to reveal which chunking strategies RAG content actually needs, well before a change ships broadly to production.

Because stored's memory is written incrementally as agents work, entities and facts, not documents to be split later, there's no separate chunking step to tune. Each memory is embedded, indexed for full-text search, and linked into the graph at the granularity it was actually written at, which is usually close to ideal for retrieval by construction.

That said, when memories do reference longer content, stored's hybrid retrieval still combines vector, keyword, and graph search over that content, so even less-than-ideal chunking on the input side gets partially compensated for by not relying on vector similarity alone.