Semantic Search: How It Works and Where It Falls Short

Semantic search converts both the query and every searchable document into vector embeddings, numeric representations positioned so that similar meaning corresponds to numeric closeness. At query time, the system finds the stored vectors closest to the query's vector and returns those as the most relevant results, regardless of exact word overlap.

This is a genuinely different mechanism from traditional search, which indexes words and matches on their presence or frequency. Semantic search doesn't care whether "cancel my plan" and "end your subscription" share any words; it only cares that a model trained on language learned they mean roughly the same thing.

The embedding models behind this rely on techniques with a long research history. Word2vec, one of the earliest widely used approaches, showed that training a model to predict nearby words from a large corpus produces vectors where semantically related terms end up close together in the resulting space, an idea modern sentence- and document-level embedding models still build on at much larger scale.

Semantic search needs somewhere to store and query millions of embeddings efficiently, which is what a vector database is built for: an index structure optimized for finding the nearest vectors to a query vector quickly, instead of the exact-match lookups a traditional database is built around. Without one, semantic search over anything beyond a small dataset becomes too slow to be useful.

The vector database you choose shapes what a natural language query actually costs at runtime: how many embeddings it can search per second, how fresh the index stays as new content is written, and how well it composes with other retrieval methods running alongside it. That's a separate decision from choosing an embedding model, and getting it wrong tends to show up first as latency, not as bad results.

Keyword search is precise and predictable: it matches what's literally there, which makes it reliable for exact terms, product codes, or names, and easy to reason about when it fails. Its weakness is paraphrasing: a synonym, a rewording, or a typo can cause a genuinely relevant result to be missed entirely.

Semantic search flips that tradeoff. It handles paraphrasing and natural language well, since it's matching meaning rather than surface form, but it can be less predictable for exact terms and sometimes surfaces results that are topically related but not actually what the user needed, a failure mode keyword search rarely produces.

Conversational, natural-language queries are semantic search's clearest win, exactly the kind of input an AI agent generates constantly. A user rarely phrases a question the same way a document answering it was written, and semantic search closes that gap without requiring either side to guess the other's exact phrasing.

It also handles synonym and terminology drift gracefully. A support ticket that says "the app crashed" and a knowledge base article about "unexpected application termination" will still match, something a purely literal search system has no way to bridge without a manually maintained synonym list.

Semantic search struggles with anything that depends on exact matching: order numbers, API keys, precise dates, or specific names that don't carry distinctive meaning on their own. It also struggles with negation and precise logical constraints, "servers that are not in production" can return production servers just fine, since the embedding captures the topic, not the logic.

It's also blind to explicit relationships between facts. Semantic search can find content related to a topic, but it can't reliably answer a question that depends on following a chain of connections between entities, the kind of question a knowledge graph is built to answer instead.

stored runs semantic search as one leg of every query, alongside keyword and graph-based retrieval, as part of hybrid retrieval. That means a natural-language question gets the benefit of semantic matching without losing the precision keyword search provides for exact terms, or the relational reasoning graph traversal provides for connected facts.

Every memory is embedded automatically the moment it's written, so semantic search is available immediately, not as a separate indexing job you have to run and maintain, and it works the same way whether the query comes from a human in the dashboard or an agent over the remote MCP endpoint.

The honest way to check whether this kind of retrieval is working is with a labeled evaluation set: a list of realistic queries paired with the specific documents or memories each one should surface, measured as recall at a fixed cutoff, the top three or five results, rather than a single overall accuracy number. Testing against a handful of hand-picked examples tends to overstate how well a system actually performs on the messier queries real users type.

Two failure patterns are worth watching for specifically. The first is topical drift: results that are thematically related to the query but don't actually answer it, since the embedding captured the general subject without capturing what was specifically being asked. The second is embedding staleness, where content written or edited after the index was built doesn't get re-embedded promptly, so freshly stored information stays invisible to a query that should find it until the index catches up.

Choice of embedding model matters more than teams often expect going in. A model trained mostly on general web text will underperform on queries full of domain-specific jargon, product names, or internal shorthand, so evaluating a candidate model against your own realistic query set, not a generic benchmark, is the only reliable way to know whether it will perform well on the content your system actually holds.

How a document gets split up before it's embedded has a bigger effect on retrieval quality than most teams expect. Embed an entire long document as one vector, and its representation ends up as an average of everything the document covers, diluted enough that a query about one narrow detail buried in it may not match well at all. Split too aggressively into single sentences, and each chunk loses the surrounding context that gave it meaning in the first place.

The practical middle ground most systems converge on is chunking by a natural boundary, a paragraph, a section, a single fact or memory, so each embedded unit is small enough to represent one coherent idea but large enough to carry the context needed to match a real query. For AI agent memory specifically, this tends to mean storing individual facts or decisions as their own units rather than embedding an entire conversation transcript as one block, which keeps each unit focused enough for a query to reliably find it.

Getting chunk size right is closely related to why over-splitting or under-splitting both quietly degrade retrieval without producing an obvious error: the system doesn't fail loudly, it just returns technically-related-but-not-quite-right results more often, which is exactly the topical drift failure mode worth testing for directly rather than assuming it isn't happening.