RAG chunking explained: chunk size, overlap, and what it costs

RAG is a simple idea. You have documents, someone asks a question, you find the relevant pieces and paste them into the prompt so the model can answer from your data.

The word doing the heavy lifting there is “pieces”. You can’t embed a whole book as one vector, and you can’t paste a whole book into a prompt. So you cut the documents into chunks. How you cut them decides how much you pay, how much you store, and whether retrieval works at all.

Let’s go through the math.

Why chunk at all?

Two reasons.

First, embeddings work on limited text. An embedding model turns a piece of text into one vector. If the piece is too big, the vector becomes a blurry average of everything in it, and retrieval gets worse.

Second, the context window. Whatever you retrieve has to fit in the prompt, next to the system prompt and the user’s question. Small chunks let you include several relevant passages instead of one giant blob.

So chunking is a trade-off. Small chunks are precise but lose surrounding context. Big chunks keep context but retrieve fuzzier and eat your prompt budget.

From documents to chunks

A chunk is measured in tokens. A useful rule of thumb: an English word is about 0.75 tokens. A 2,000-word document is about 1,500 tokens.

Chunking works as a sliding window. You take the first chunk size tokens, then step forward and take the next window. If you use overlap, each window repeats the tail of the previous one, so a sentence sitting on a boundary isn’t cut in half.

The chunk count per document:

step = chunk size − overlap tokens
chunks = 1 + ceil((doc tokens − chunk size) ÷ step)

Let’s work an example. 1,000 documents, 2,000 words each. That’s 1,500 tokens per document. Chunk size 512, overlap 15% (so 76 tokens of overlap, step 436):

chunks per doc = 1 + ceil((1500 − 512) ÷ 436) = 4
total chunks = 1000 × 4 = 4000

Notice what overlap does. With no overlap the step is 512 and you get 3 chunks per document. At 15% you get 4. At 50% the step halves and you get 5. Overlap is duplicated text: you embed it, you store it, you pay for it twice. That’s why you rarely see more than 15–20% in practice.

The embedding cost

Every chunk gets embedded once, and providers charge per token:

embed tokens = total chunks × chunk size
cost = embed tokens ÷ 1M × price per 1M

Our 4,000 chunks at 512 tokens each:

4000 × 512 = 2,048,000 tokens

At mid-2026 prices:

OpenAI text-embedding-3-small ($0.02/M):  $0.04
Voyage voyage-3 ($0.06/M):                $0.12
Cohere embed-v4 ($0.10/M):                $0.20
OpenAI text-embedding-3-large ($0.13/M):  $0.27

Cents. Embedding cost is almost never the problem, even at 100× this corpus size. What people actually get wrong is the next two parts.

The storage cost

Each chunk becomes a vector of floats. The size depends on the model’s dimensions:

bytes per vector = dimensions × 4

text-embedding-3-small produces 1536 dimensions, so 6 KB per vector. Our 4,000 chunks:

4000 × 1536 × 4 = 23.4 MB

Plus roughly 512 bytes of metadata per chunk (id, source, offsets), another 2 MB. Total: about 25 MB.

Also small. But dimensions matter: text-embedding-3-large is 3072 dimensions, double the storage for every chunk, forever. If you’re paying a vector database per GB, the embedding model you pick is a permanent storage decision, not just a quality one.

Does the retrieval fit?

Here’s the part that actually breaks RAG systems. At query time you retrieve the top-k most similar chunks and paste them into the prompt:

retrieval tokens = k × chunk size

That has to fit in the context window, next to everything else:

retrieval tokens ≤ context window − system prompt − question

With chunk size 512, k=5, an 8K window (8,192 tokens), a 500-token system prompt and a 100-token question:

5 × 512 = 2560 tokens
8192 − 600 = 7592 available

Fits, with room for the answer.

Now bump chunk size to 1,024 and k to 8:

8 × 1024 = 8192 tokens

That’s the entire window before the system prompt even enters. It doesn’t fit. And this failure is quiet: most stacks silently truncate, and you spend a week wondering why answers got worse.

I built a calculator that runs all of this at once, chunk count, embedding cost per provider, vector storage, and the context-window check: the RAG chunking calculator.

Picking values

My advice, as defaults to start from:

• Chunk size 512 tokens. Big enough to hold a coherent passage, small enough to retrieve precisely.
• Overlap 10–15%. Enough to protect boundaries, cheap enough to not care about.
• k = 5, then check the math against your context window before raising it.

Then tune based on your documents. Dense reference docs like smaller chunks. Narrative content tolerates bigger ones.

One last thing. All of this is deterministic arithmetic, and it should stay that way: compute the chunk counts and budgets in code, don’t ask the model to reason about them. I wrote about why in don’t let the LLM do the math. And if you’re running the generation side yourself, the context window you pick also affects memory: see how much VRAM you need to run an LLM locally and the local vs API cost math.

Related posts about ai: