Embeddings, vector stores, and RAG

Imagine managing 12,000 customer support articles totaling over 40 million , while your language model operates with a 200,000-token context window. Even with modern long-context models, you cannot simply dump the entire knowledge archive into a single prompt. An external system must pinpoint the handful of articles relevant to a user's question and deliver those excerpts to the model at query time.

The infrastructure that solves this search-and-assembly challenge—and the question of when to use retrieval versus simply expanding context windows—is the focus of this guide.

Vector embeddings

An is a high-dimensional vector—a fixed-length array of floating-point numbers, commonly 1,536 or 3,072 dimensions—that represents the conceptual meaning of a passage of text. To a human reader, the raw coordinates are unintelligible. Mathematically, however, their spatial orientation reflects semantic relationships: embedding models are trained so that passages with similar meanings map close together in vector space, while unrelated topics land far apart.

Generating embeddings requires an embedding model, which is distinct from generative language models. Embedding models are compact, fast, and inexpensive to run. Instead of generating conversational prose, they take an input string and output its coordinate vector. In a production retrieval architecture, a lightweight model handles vectorization, while a much larger frontier model synthesizes the final answer—and the two models often come from different providers.

Two operational characteristics of embeddings matter in system design:

1. Every embedding generated by a given model has identical dimensionality regardless of input length. A three-word title and an 800-word excerpt both produce a vector of the exact same size.
2. Embeddings from different models inhabit incompatible vector spaces. If you migrate from one embedding model to another, you must re-embed your entire document collection from scratch.

Strengths and pitfalls of semantic search

Measuring the similarity between two vectors—typically using cosine similarity or dot product—enables . Rather than matching exact keywords, semantic search identifies conceptual matches. Searching for "stopping a repeating payment" easily retrieves an article titled "cancel a subscription", even though the two phrases share zero vocabulary.

While powerful, semantic similarity has well-known blind spots that often catch engineering teams off guard:

• Semantic proximity of antonyms: Phrases like "payment was approved" and "payment was declined" discuss the exact same subject matter, use identical vocabulary, and appear in similar contexts. An embedding model naturally clusters them close together because it captures topical domain rather than truth value. As a result, searching for approvals will routinely return decline notices.
• Divergent domain vocabulary: A customer might search for "my checkout was rejected", while the official troubleshooting document is titled "issuer decline code 05". Unless the embedding model was exposed to that domain mapping during training, the vector distance between the layman's description and the technical manual may be substantial.
• Exact identifiers and technical strings: Specific product codes like AX-4471-B, order IDs like 90183, or error strings like ECONNRESET perform poorly under pure semantic search. Vectorization projects discrete identifiers into generalized semantic neighborhoods, destroying the exact character matching required to locate a unique record.

Vector stores and indexes

A is a database optimized to store high-dimensional embeddings alongside source documents and execute nearest-neighbor queries against incoming prompts. Options range from dedicated vector databases to vector extensions in relational databases like PostgreSQL (pgvector), as well as fully managed cloud knowledge services.

While comparing a query vector against every stored document via brute-force linear scan is mathematically precise, it quickly becomes unviable at scale. Calculating cosine distances against 60,000 document chunks on every query introduces noticeable latency. Modern vector stores use Approximate Nearest Neighbor (ANN) indexing algorithms (such as HNSW or IVF) to organize vectors into traversable clusters. This delivers sub-10ms search performance across millions of vectors, accepting a slight probability of missing a true nearest neighbor in exchange for massive latency gains.

In industry discussions, the combination of source documents, vector embeddings, storage indexes, and ingestion pipelines is commonly called a .

Chunking strategies

Documents cannot simply be embedded whole. Converting an entire 50-page employee handbook into a single vector averages out fifty pages of distinct policies into a generic coordinate that matches nothing accurately. Before embedding, documents must undergo —breaking large texts into discrete, coherent segments.

Two core parameters govern chunking:

• Chunk size: The volume of text contained in each slice. Chunks that are too small lack the surrounding context needed to interpret their meaning, while oversized chunks dilute specific facts with surrounding prose.
• Chunk overlap: The number of tokens shared between adjacent chunks. Overlap ensures that key concepts or sentences that fall across a split boundary are not cut in half.

Consider a practical example from an e-commerce refund policy:

Refunds are processed within 5 business days of approval.

This timeline does not apply to marketplace orders, which are settled directly by third-party sellers and can take up to 21 days.

If an arbitrary chunk split cuts cleanly between these two paragraphs without overlap, the first chunk becomes a misleading, quotable statement for marketplace buyers. A retrieval query matches the first chunk, places it in the prompt, and the generative model has no visibility into the seller exception. Setting adequate chunk overlap or splitting along logical section headers prevents these boundary errors.

Setting chunk size

Cut on a boundary the document already has.

1. Split on headings, sections, or list items first. Split on token count only where no boundary exists.
2. Set the overlap to at least the length of one sentence.
3. Keep a heading and its first paragraph in the same chunk.
4. Store the document title and section with every chunk.
5. Test the split by retrieving chunks for ten real questions. Read the chunks.
6. Re-embed the whole corpus after you change the chunk size.

The complete retrieval-augmented generation pipeline

When a user submits a question, the vector store queries for the most similar chunks, returning a quantity known as . Requesting $k=3$ chunks returns three passages. Requesting $k=50$ floods the prompt with noise, undermining the precision retrieval was meant to provide.

To balance recall with precision, production architectures combine vector search with two complementary techniques:

First, combines vector similarity with traditional full-text keyword search (such as BM25). Keyword search guarantees that specific error codes, part numbers, and method names match exactly, while semantic search captures conceptual synonyms. The system merges both result sets into a balanced candidate pool.

Second, evaluates candidate chunks with a cross-encoder model. While bi-encoder embedding models compute query and document representations independently, a reranker evaluates the query and candidate chunk together, scoring direct relevance. The system can retrieve fifty broad candidates via fast hybrid search, run them through a reranker, and pass only the top three highest-scoring excerpts to the generative model.

Putting these steps together produces (RAG):

The final answer generated in step 9 depends entirely on the quality of retrieved passages. If the relevant policy was missed during retrieval, prompt engineering cannot recover the fact. When debugging incorrect RAG answers, always examine the retrieved passages first.

The window is a fixed-size hole. Retrieval is the filter in front of it, and the only question it answers is which three documents go through.

When RAG outperforms large context windows

With modern models supporting context windows of one to two million tokens, it is natural to ask whether complex RAG pipelines are still necessary. In practice, RAG provides several key advantages:

• Vast corpus scales: A million tokens equals roughly 3,000 pages of text. Enterprise document archives, codebases, and customer records frequently run into hundreds of millions of tokens, far exceeding any available context window.
• Cost efficiency across conversational turns: Every token placed in a context window is billed on every conversational turn. If an application pastes 300,000 tokens of documentation into the prompt on a 15-turn chat, it bills 4.5 million input tokens. Dynamically retrieving 4,000 relevant tokens per turn keeps input costs negligible by comparison.
• Data freshness and live updates: Information in a context window is fixed when the request is sent. A RAG pipeline queries live databases at the exact moment a question is asked, instantly reflecting updates made minutes earlier without requiring application restarts or cache invalidation.
• Verifiable citations and provenance: RAG pipelines track the source URL, document ID, and chunk metadata for each retrieved passage, enabling applications to generate trustworthy, clickable source citations for users.
• Granular access control and permissions: Enterprise systems require document-level access control. RAG pipelines can filter document candidates based on the active user's permissions before content reaches the model prompt. Once sensitive text is injected into an LLM context, there is no reliable way to prevent the model from revealing it in subsequent answers.
• Attention focus: Models reason more effectively over a compact, highly relevant prompt than over a million-token document dump containing competing information.

When direct context stuffing is preferable

Building and maintaining embedding pipelines, vector stores, and rerankers introduces architectural complexity. Loading content directly into the context window is the simpler and superior choice in three common scenarios:

1. The dataset easily fits: If a task involves forty documents totaling 80,000 tokens, building an indexing pipeline is unnecessary overhead. Placing the documents directly into the prompt and leveraging prompt caching provides instant, accurate answers with minimal complexity.
2. Whole-document synthesis: Questions like "Compare the liability terms in these three vendor contracts" or "Identify all internal contradictions across this specification" require cross-document synthesis. Fixed chunk retrieval often misses cross-cutting patterns.
3. Structured or deeply relational assets: Relational database schemas, spreadsheets, or interconnected source code trees lose essential structural meaning when chopped into arbitrary 800-token chunks. Supplying the full structure intact allows the model to reason across relational boundaries.

Modern production architectures increasingly combine both patterns: they use retrieval to filter thousands of documents down to five or six relevant files, and then inject those selected files in their entirety into a large context window.

Terms introduced

• Embedding: a fixed-length list of numbers standing for a piece of text, positioned so that similar meanings land near each other.
• Vector store: a database that holds embeddings beside their source text and answers nearest-neighbour queries against them.
• Semantic search: finding text by distance between embeddings rather than by matching words.
• Retrieval-augmented generation: retrieving relevant text from a corpus and putting it in the context window before asking the model to answer.
• Chunking: cutting documents into pieces before embedding them, governed by a chunk size and an overlap.
• Top-k: how many nearest chunks a search returns.
• Reranking: scoring retrieved candidates with a second model that reads the query and the chunk together, then keeping the best few.
• Hybrid search: combining keyword matching and vector search over the same corpus, so exact identifiers and paraphrases both work.
• Knowledge base: the documents, their embeddings, the store holding them, and whatever keeps the three in step.

---

How providers do it

Both vendors sell a managed retrieval service that hides the embedding model, the store and the chunking behind one tool. The difference is how much of the pipeline you can see and change.

Concept	OpenAI	Google
Embedding models	text-embedding-3-small 1,536, text-embedding-3-large 3,072	gemini-embedding-2 and gemini-embedding-001, 3,072
Reduce dimensions	dimensions parameter, v3 models only	output_dimensionality, 128 to 3,072
Managed retrieval	Vector stores plus the file_search tool	File Search in the Gemini API; RAG Engine on Vertex AI
Chunk size default	800 tokens, overlap 400	File Search defaults not stated; RAG Engine 1,024 with 256 overlap
Hybrid search	Yes, inside file search ranking options	Yes, in Vector Search. Not documented for File Search
Reranking	No standalone model; ranking happens inside file search	Ranking API semantic ranker, plus an LLM reranker in RAG Engine
Storage cost	1 GB free, then $0.10 per GB per day	Free; embeddings billed at indexing time

Every row is confirmed against the docs read on the date below, except the two Google File Search rows on chunking defaults and reranking, which the vendor docs do not state.

The row that decides an architecture is reranking. Where the platform has no separate reranker, a pipeline that needs one has to add a model of its own, and that is a second vendor in the query path.

OpenAI

What this maps to: the Embeddings API for the vectors, managed vector store objects for the index, and a file_search tool on the Responses API for the query side.

Embeddings

Question	Answer	Status
Which models are offered?	text-embedding-3-small at 1,536 dimensions, text-embedding-3-large at 3,072, and the legacy text-embedding-ada-002 at 1,536	confirmed
Can the dimensionality be reduced?	Yes on the v3 models, with a dimensions parameter. The docs give a 3-large embedding shortened to 256 still outperforming an unshortened ada-002 at 1,536. Not available on ada-002	confirmed
Maximum input?	8,192 tokens	confirmed
Price per million tokens?	3-small $0.02, 3-large $0.13, ada-002 $0.10	confirmed

Vector store and file search

Question	Answer	Status
Is there a managed store?	Yes. Vector store objects hold uploaded files, and the file_search tool on the Responses API queries them	confirmed
Is it semantic only?	No. The tool is described as retrieving "through semantic and keyword search", and the retrieval guide describes hybrid search balancing embedding and text weights in the ranking options	confirmed
How is chunking configured?	max_chunk_size_tokens, default 800, and chunk_overlap_tokens, default 400. The chunk maximum is 4,096 tokens and the minimum 100. Overlap may not exceed half the chunk size	confirmed
Can results be filtered by score?	Yes. Ranking options include a score_threshold from 0.0 to 1.0	confirmed
What are the ranker's exact option values?	Names such as hybrid_search.embedding_weight and a dated ranker identifier were seen only in search snippets, not on a vendor page	unconfirmed
Is there a standalone reranker model?	None found. There is no /rerank endpoint and the reranking guide URL returns 404. Reranking appears only inside file search	confirmed
What does storage cost?	The first 1 GB is free, then $0.10 per GB per day on parsed content. Tool calls are $2.50 per 1,000	confirmed
Are these prices current?	Rates move. Read the pricing page before quoting them	unconfirmed

Their vocabulary

Standard term	Their term
Vector store	Vector store
Retrieval tool	file_search
Chunk size and overlap	max_chunk_size_tokens, chunk_overlap_tokens
Hybrid search	Hybrid search, tuned by ranking options

Where to look

The vector store object's file counts show what actually indexed and what failed to parse, which is the first place to look when a document is never retrieved. Expiration policies on a store are the lever on storage cost.

Last verified: 2026-09-09 against the embeddings guide (https://developers.openai.com/api/docs/guides/embeddings), the retrieval guide (https://developers.openai.com/api/docs/guides/retrieval), the file search guide (https://developers.openai.com/api/docs/guides/tools-file-search) and the pricing page (https://developers.openai.com/api/docs/pricing).

Google

What this maps to: several products at different levels. File Search in the Gemini API is managed end to end. Vertex AI Vector Search is the index on its own. Vertex AI RAG Engine assembles the pipeline. Grounding with Google Search retrieves from the public web instead of a corpus.

Embeddings

Question	Answer	Status
Which models are offered?	gemini-embedding-2 and gemini-embedding-001, with gemini-embedding-2-preview also listed	confirmed
What dimensionality?	Both output 3,072 dimensions by default	confirmed
Can it be reduced?	Yes, with output_dimensionality, supported from 128 to 3,072	confirmed
Is there a task hint?	task_type exists on gemini-embedding-001 with values including RETRIEVAL_DOCUMENT, RETRIEVAL_QUERY, SEMANTIC_SIMILARITY and CODE_RETRIEVAL_QUERY. It cannot be used with gemini-embedding-2, where task instructions go in the prompt instead	confirmed

Retrieval products

Question	Answer	Status
Is there a managed store in the Gemini API?	Yes. File Search imports, chunks and indexes files into a store, and a FileSearch tool queries it	confirmed
How is chunking configured?	chunking_config.white_space_config.max_tokens_per_chunk and max_overlap_tokens. The guide's example uses 200 and 20	confirmed
What are the chunking defaults?	The reference says defaults exist and never states them	unconfirmed
What does File Search cost?	Charged for embeddings at indexing time. Storage is free, query-time embeddings are free, and retrieved chunks bill as ordinary context tokens	confirmed
Does File Search rerank or do hybrid search?	The docs describe embedding search only and are silent on both	unconfirmed
What is the index product on Vertex AI?	Vector Search, built on ScaNN, supporting dense, sparse and hybrid search	confirmed
What assembles a full pipeline?	Vertex AI RAG Engine, now described as part of the Gemini Enterprise Agent Platform, with backends including its managed database, Vector Search, Feature Store, Weaviate and Pinecone	confirmed
What are RAG Engine's chunking defaults?	chunk_size 1,024 tokens and chunk_overlap 256 tokens	confirmed
Is there a reranker?	Yes, two. The Ranking API's semantic ranker, semantic-ranker-default@latest and semantic-ranker-fast@latest, stateless and documented as sub-100ms; and an LLM reranker in RAG Engine costing an extra Gemini call at 1 to 2 seconds	confirmed
Is there retrieval from the open web?	Yes. Grounding with Google Search, enabled with {"type": "google_search"} in tools, returns inline citations to source URLs	confirmed

Their vocabulary

Standard term	Their term
Managed retrieval in the Gemini API	File Search, with a File Search store
Vector index	Vector Search
RAG pipeline	RAG Engine
Reranking	Ranking API, semantic ranker, or LLM reranker
Web retrieval	Grounding with Google Search

Where to look

RAG Engine exposes the chunking transformation settings directly, so the chunk size and overlap that decide retrieval quality are visible rather than implied. For File Search, the store's import result is where a document that never gets retrieved will show up as a parse failure.

Last verified: 2026-09-09 against the embeddings guide (https://ai.google.dev/gemini-api/docs/embeddings), the File Search guide (https://ai.google.dev/gemini-api/docs/file-search), the RAG Engine transformation docs (https://docs.cloud.google.com/vertex-ai/generative-ai/docs/rag-engine/fine-tune-rag-transformations), the Vector Search overview (https://docs.cloud.google.com/vertex-ai/docs/vector-search/overview) and the Ranking API docs (https://docs.cloud.google.com/generative-ai-app-builder/docs/ranking).

---

Check your understanding

0 of 4 answered

1. What produces the embeddings a vector store holds?

   A separate, smaller embedding model, run over each chunk before it is stored.The same model that answers the question, on its first call.The vector store, which computes them from the text it is given.The tokenizer, as a byproduct of splitting text into tokens.
2. A support search built on pure vector similarity fails on queries containing product codes like AX-4471-B. Why, and what fixes it?

   An embedding turns an identifier into an approximate region of meaning, so the exact string is lost. Hybrid search adds keyword matching alongside it.The codes were never embedded. Re-embedding the corpus fixes it.The chunk size is too large. Smaller chunks make identifiers easier to match.Vector stores cannot index text containing digits or hyphens.
3. A corpus of 40 internal documents totals 90,000 tokens, every user may read all of them, and questions often require comparing two documents. Retrieval or a big window?

   Put them all in the window and cache the prefix. Retrieval adds machinery and would break the comparisons.Retrieval, because a knowledge base is always the more scalable design.Retrieval, because 90,000 tokens will not fit in any current window.Neither. A corpus this size has to be fine-tuned into the model.
4. A refunds policy states a 5 day rule in one paragraph and an exception for marketplace orders in the next. Chunking splits between them with no overlap. What is the result?

   Retrieval returns a chunk that is confidently wrong for marketplace customers, and nothing in the window contradicts it.The exception is retrieved instead, because it is the more specific text.Both chunks are always retrieved together, because they came from one document.The model notices the missing context and asks a clarifying question.