Embeddings and Vector Databases Explained

Keyword search does not understand meaning. It matches strings. Ask it for “car” and it will miss every document that says “vehicle” or “automobile.” Embeddings fix that. They turn text into numbers that capture meaning, and vector databases let you search those numbers at scale. This is the machinery behind semantic search, recommendations, and most retrieval-augmented AI apps shipping today.

If you plan to build anything that answers questions over your own data, you will meet these two tools fast. Here is how they actually work and how to use them without wasting weeks.

What an embedding actually is

An embedding is a list of numbers that represents a piece of text. A short sentence might become a list of 768 or 1536 numbers. That list is called a vector. Each vector is a point in a high-dimensional space, and the position of that point encodes meaning.

The key property is simple: similar meanings land close together. “How do I reset my password” and “I forgot my login” end up near each other, even though they share almost no words. “How do I bake bread” lands far away. You never read these numbers yourself. You let a model produce them and you measure the distance between them.

An embedding model does the conversion. You send text, you get a vector back. The same model must be used for everything you want to compare, because two different models produce vectors that live in different spaces and cannot be compared. Pick one model per project and stick with it.

Why you need a vector database

Once your text is a set of vectors, search becomes a math problem: given a query vector, find the stored vectors closest to it. With a few hundred items you can compare against every one of them in a loop. That is called brute force, and it works fine at small scale.

It stops working when you have millions of vectors. Comparing a query against ten million points on every request is slow and expensive. A vector database solves this with an index built for approximate nearest neighbor search. It does not check every vector. It uses a smart structure to jump close to the right neighborhood, then refines. You trade a tiny amount of accuracy for enormous speed.

A good vector database also handles the boring but critical parts: storing metadata next to each vector, filtering by that metadata, updating and deleting records, and staying fast as data grows. That plumbing is why you reach for a dedicated tool instead of writing your own loop.

How similarity is measured

Search comes down to a distance or similarity score between two vectors. Three measures show up most often.

• Cosine similarity looks at the angle between two vectors and ignores their length. It is the default for text and the one you will use most.
• Dot product factors in both angle and magnitude. Some models are trained to work best with it, so check what your model recommends.
• Euclidean distance measures straight-line distance between the two points. Common with image and spatial data.

The rule that matters: use the metric your embedding model was trained for. Mixing a model tuned for cosine with a dot-product index quietly degrades your results, and the failure is silent. Nothing crashes. Answers just get worse.

A minimal pipeline that works

Most production systems follow the same shape, and you can build the first version in an afternoon.

• Chunk your content. Split documents into pieces of a few hundred words. Whole documents make vectors too vague. Single sentences lose context. Aim for coherent paragraphs.
• Embed each chunk. Run every chunk through your embedding model and get a vector back.
• Store vectors plus metadata. Save each vector with its source, title, and any tags you will want to filter on later.
• Embed the query at search time. When a user asks something, embed their question with the same model and ask the database for the closest chunks.
• Feed results to your model or UI. For an AI app, hand the top chunks to a language model as context so it answers from your data. This pattern is retrieval-augmented generation, and embeddings are the retrieval half of it.

That is the entire loop. Everything else is tuning.

Mistakes that quietly wreck results

The failures here are rarely loud. The system returns something, it just returns the wrong thing. Watch for these.

Chunks that are too big or too small. This is the single biggest lever on quality. If retrieval feels random, resize your chunks before you touch anything else.

Mixing embedding models. Re-embedding half your data with a newer model and leaving the rest breaks every comparison across the boundary. If you upgrade the model, re-embed everything.

Skipping metadata filters. Pure vector search ignores hard constraints. If a user should only see their own documents, filter by owner in the database, not after. Combine semantic search with plain filters.

Ignoring cost at scale. Embedding millions of chunks and re-embedding on every content change adds up. Cache vectors, only re-embed what changed, and pick vector dimensions you can afford to store and search.

Do you even need one yet

Be honest about scale. If you have a few thousand records, brute-force similarity in memory or a simple database extension is often enough and far simpler to run. Reach for a dedicated vector database when data grows into the hundreds of thousands or millions, when you need sub-second search, or when you need filtering, live updates, and reliability under load.

Do not over-engineer the first version. Ship the simple loop, watch real queries, then scale the infrastructure once you see where it strains. Premature vector databases are as wasteful as premature microservices.

Getting started

Start tiny. Take one document set you know well, chunk it, embed it, and run ten real questions against it. You will learn more from that hour than from a week of reading. Once retrieval feels sharp on a small set, the same pipeline scales to millions of records with the right database behind it.

If you want to go deeper, our guide on retrieval-augmented generation shows how embeddings feed a language model, and our piece on building an AI knowledge base walks through a full setup end to end.

Want a second pair of eyes on your architecture before you build? Message our team in the Neurounit bot and we will help you scope it right.