Hybrid Search: Combining Vector, Keyword, and Graph Retrieval

In its most common form, hybrid search runs three retrieval methods against a query in parallel: vector similarity search for semantic matches, full-text search for exact keyword and phrase matches, and, in graph-backed memory systems, graph traversal for entities and their explicit relationships. Each returns its own ranked candidate list.

The specific combination varies by system, some pair only vector and keyword search, others add graph traversal as a third signal, but the underlying logic is the same: no single method is asked to catch everything a query might need, because no single method reliably can.

Recent research on retrieval-augmented systems found that retrieval failures are common even in well-built pipelines, and that different failure modes call for different fixes, a strong signal that no single retrieval strategy is sufficient on its own. Vector search alone drops precise facts. Keyword search alone drops paraphrases and synonyms. Graph search alone drops anything that hasn't been structured into entities yet.

The failures also aren't rare edge cases. In real usage, a single session commonly needs an exact fact, a semantically related concept, and an explicit relationship, sometimes in the same query. A system tuned to only one retrieval style will reliably handle a third of that and miss the rest.

Running three retrieval methods produces three ranked candidate lists that need to become one. The simplest approach merges by a weighted score across methods; more sophisticated systems apply a reranking model, a smaller model trained specifically to judge relevance, over the combined candidate set to produce a final order that reflects true relevance better than any single method's raw ranking.

This merge-and-rerank step matters more than it might seem. A hybrid search that just concatenates three lists without a real merge strategy tends to surface duplicates and irrelevant results near the top, which defeats the purpose. The value of hybrid search comes from the ranking quality of the combined result, not just from running more queries.

Classic retrieval-augmented generation pipelines are often built around vector search alone, since it's the simplest retrieval method to stand up. Hybrid search is what those pipelines evolve into once teams notice specific, reproducible gaps: an exact ID that never gets found, a relationship the model can't reason about because it was never modeled explicitly.

The practical upgrade path is usually incremental: add keyword search alongside vector search first, since it's the cheapest gap to close, then add graph-based retrieval once the underlying content is structured enough to support it. Each addition covers a specific class of query the previous setup was missing.

stored's memory graph runs all three retrieval strategies, graph traversal, vector similarity, and full-text search, on every query by default, merged into one ranked result. On paid plans, a premium reranker takes a second pass over the merged candidates, so the most relevant memories consistently surface first regardless of which retrieval method originally found them.

That's the difference between hybrid search as a concept and hybrid search as something you can actually rely on: it's not an option you configure, it's how every query into stored's memory behaves by default, whether the answer needs an exact fact, a related concept, or a chain of relationships.

Not every system that claims to run hybrid search actually merges results well. The most useful way to evaluate one is with a labeled test set: a fixed list of realistic queries paired with the results that should come back, so you can measure recall at a given cutoff (did the right memory appear in the top five results) rather than eyeballing a handful of example queries and assuming the pattern holds.

Latency matters just as much as quality. Running vector similarity, keyword search, and graph traversal in parallel should add only a small amount of wall-clock time over any one method alone; if adding a second or third retrieval strategy roughly multiplies response time, the implementation is running the methods sequentially instead of concurrently, which defeats much of the point of combining them in the first place.

The most frequent mistake is treating the merge step as simple concatenation: take the top results from each method, stack the lists, and call it hybrid search. Vector similarity scores and keyword relevance scores live on entirely different scales, so combining them without normalization or a proper reranking step tends to produce a final order that's dominated by whichever method happens to produce larger raw numbers, not whichever result is actually most relevant.

A second common mistake is adding graph traversal before the underlying content is structured enough to support it. Graph-based retrieval depends on entities and relationships already being modeled explicitly; bolting it onto unstructured text too early adds complexity without adding coverage, since there's nothing for the traversal step to actually walk. The practical fix is sequencing: get vector and keyword search merged and reranked well first, then add graph traversal once there's real entity structure behind it.

Hybrid search isn't the right starting point for every system. A small, well-defined document set with predictable, mostly literal queries, an internal glossary or a short FAQ, often does fine on keyword search alone, and adding vector or graph retrieval on top just adds infrastructure and latency without a matching gain in result quality.

The signal to actually build hybrid search is a recurring pattern of queries that one method demonstrably misses: paraphrased questions a keyword index can't match, exact identifiers a vector index blurs together, or relationship questions neither can answer. Building it in response to that evidence, rather than by default, keeps the system as simple as the query patterns actually require.

For an AI agent's memory specifically, that evidence tends to show up fast. An agent fields exact names, IDs, and dates in the same session where it also has to recall a general preference or trace a relationship between two facts, so the mixed query pattern that justifies combining retrieval methods is closer to the default case than the exception once an agent is doing real, varied work.

This is also why most teams don't design the merge logic themselves from scratch. A managed memory layer that already runs several retrieval strategies in parallel and reranks the combined output saves the considerable engineering time it takes to get score normalization, deduplication, and latency right, time that's easy to underestimate until a first naive implementation ships and surfaces duplicate or poorly ordered results in production.