What is Chunking?
Chunking is the process of splitting documents into smaller passages before they are embedded and indexed, so a retrieval system can return the specific piece of text that answers a question instead of an entire document.
At a glance
- Documents are split into passages (chunks) before embedding — retrieval returns chunks, not whole files.
- Typical production chunks are 300–800 tokens with 10–20% overlap so ideas aren't cut in half.
- Structure-aware chunking (by heading, section, table or slide) beats naive fixed-length splitting.
- Contextual chunking prepends a short document summary to each chunk so it stands alone.
- Chunking quality sets the ceiling on retrieval quality — no reranker rescues badly split text.
Why chunking matters
A retrieval system can only return what it indexed. If you embed an entire 40-page policy document as one vector, that vector is an average of forty pages of ideas — too blurry to match a specific question, and far too long to paste into a prompt. Chunking breaks the document into passages small enough to be semantically focused and cheap to include in context.
Chunk too small and you strip away the context a passage needs to make sense ("it must be renewed annually" — what must?). Chunk too large and each vector blends multiple topics, diluting the match and burning tokens. Most production systems land between 300 and 800 tokens with a 10–20% overlap between neighbouring chunks.
Strategies that work in production
Structure-aware splitting is the first upgrade over fixed-length windows: split on headings, sections, list items, table rows or slides so each chunk is a coherent unit. Semantic chunking goes further, using embedding similarity between sentences to place boundaries where the topic actually changes.
Contextual chunking (popularised by Anthropic's contextual retrieval work) prepends a one- or two-sentence description of the parent document to every chunk before embedding it, so an isolated passage still carries who, what and when. Pair this with rich metadata — source, section, author, effective date, access-control tags — because that metadata is what enables filtering later.
Finally, plan for change: documents get updated and deleted. Give every chunk a stable ID tied to its source so you can re-index or purge a document without rebuilding the whole index.
Frequently asked questions
What is the best chunk size for RAG?
There is no universal number, but 300–800 tokens with 10–20% overlap is the common production range. Short, factual content (FAQs, product records) works well at the low end; narrative or technical prose usually needs the high end so an idea isn't split across chunks.
Why do chunks need overlap?
Overlap means consecutive chunks share a sentence or two at their boundary, so an answer that straddles a split still appears complete in at least one chunk. Without overlap, retrieval frequently returns a passage that stops mid-explanation.
What is contextual chunking?
Contextual chunking prepends a short, model-generated summary of the parent document (and often the section) to each chunk before embedding. The chunk then carries enough context to be understood — and matched — on its own, which measurably reduces retrieval failures.
Should I chunk tables and PDFs the same way as prose?
No. Tables should be kept intact (or serialised row by row with the header repeated), and PDFs should be parsed to structured text first. Naively splitting a table or a two-column PDF by character count produces chunks that no embedding model can interpret.
Putting Chunking to work?
We help data & AI teams design and ship this in production. Tell us what you're building and we'll point you at the shortest path.
Talk to our team