Vector Embeddings for AI Agents, Explained

A vector embedding is the output of an embedding model given a piece of input, usually a few hundred to a few thousand numbers, each one a coordinate in a high-dimensional space. The model is trained so that inputs with similar meaning produce vectors that are close together in that space, and inputs with different meaning produce vectors that are far apart.

This idea traces back to earlier word embedding techniques that mapped individual words to vectors based on the contexts they appeared in. Modern embedding models extend the same principle to full sentences and paragraphs, capturing meaning at the level most retrieval systems actually need.

Once text is a vector, comparing meaning becomes comparing geometry. The most common measure is cosine similarity: the angle between two vectors, regardless of their length, where a smaller angle means more similar meaning. Two sentences with nearly the same meaning will produce vectors with a small angle between them, even if the exact wording is completely different.

That's the property that makes vector search useful for retrieval: a query like "how do I reset a password" and a stored memory saying "password reset requires the account email" will land close together in vector space even though they don't share many exact words, something keyword search alone would miss entirely.

Embedding models vary in dimensionality, cost, and what kind of content they were trained to represent well. General-purpose models, like OpenAI's embedding models, work well across most text and are the common default. Domain-specific models can outperform general ones on narrow content, code, legal text, but that gain rarely justifies the added complexity for most teams.

Whatever model you choose, consistency matters more than the specific choice: every vector in an index has to come from the same model, or the geometry breaks and similarity scores stop being meaningful. Switching embedding models later means re-embedding everything, not just the new content.

The two terms sit at different layers and it's worth being precise about which is which. Vector embeddings are the numeric representations themselves, the list of numbers an embedding model produces for a given piece of text. A vector database is the storage and indexing system built to hold a large collection of those embeddings and answer nearest-neighbor queries against them quickly, at scales where comparing a query vector against every stored vector one by one would be far too slow.

You need both to build a working retrieval system: embeddings without an index don't scale past a small, in-memory collection, and an index without embeddings has nothing to search. Most teams don't think about this distinction until they're debugging a slow query, at which point it usually becomes clear whether the problem is the embeddings themselves, the wrong model, poorly chunked input, or the index sitting underneath them. Knowing which layer is at fault, before you start tuning, saves a lot of wasted effort chasing the wrong fix.

Higher-dimensional vector embeddings can capture more nuance, but that nuance isn't free. Every dimension adds to storage cost, memory footprint, and the time a similarity search takes, since more dimensions mean more arithmetic per comparison. A model producing 3,072-dimension vectors will cost noticeably more to store and query at scale than one producing 768-dimension vectors, even if the accuracy gain between them is modest for a given use case.

Some embedding models let you truncate their output to a shorter vector with a small, predictable accuracy tradeoff, which is a reasonable way to cut storage and query cost once you've measured that the shorter vector still meets your recall bar on real queries. The right dimensionality is an empirical question, not a default to accept unquestioned, and it's worth testing on your own content rather than assuming the largest available model is automatically the best choice.

The same logic applies to the model itself, not just its output size. A newer, larger embedding model usually scores better on public benchmarks, but benchmark gains don't always translate into a measurable improvement on your specific content and queries. Running a small, controlled comparison, same documents, same queries, different models, before committing to one at scale is cheap insurance against a costly re-embedding project later if the choice turns out to be a poor fit.

Vector embeddings capture semantic similarity, not factual precision. They're good at finding text about the same topic and bad at finding an exact match, an ID, a specific number, a precise date, since those don't reliably embed in a way that distinguishes them from nearby-but-wrong values. A query for order number 48213 can return content about order numbers in general without ever finding the specific one.

This is the core reason retrieval systems built on embeddings alone tend to plateau: they're excellent at fuzzy, topical recall and weak at exact and relational lookups. Most production systems pair vector search with keyword and graph-based retrieval specifically to cover that gap, an approach usually called hybrid retrieval.

There's a second, subtler limit worth knowing about: embeddings drift with context length. A vector produced for a single sentence and a vector produced for a ten-paragraph chunk containing that sentence won't be equally useful for the same query, since the longer chunk's embedding represents an average of everything in it, diluting any one specific fact. That's part of why chunking strategy matters as much as model choice: an embedding can only be as precise as the span of text it was computed over.

Every memory written to stored is embedded automatically as part of the write, so it's immediately searchable by semantic similarity, no separate embedding pipeline to build or maintain. stored also supports bringing your own OpenAI key, encrypted at rest, so embedding costs stay on your own account rather than being marked up.

Because embeddings are only one leg of stored's hybrid retrieval, a query that needs an exact fact or a relationship still gets answered correctly, the graph and keyword indexes cover what vector similarity alone would miss, all merged into a single ranked result.