A RAG system answers questions from your company documents. First, it cuts each document into small pieces called chunks, and it searches those pieces when someone asks a question. If the cuts lose context, the AI model never sees the right passage. In Anthropic’s tests, standard embedding search missed about 6 in every 100 relevant passages among the 20 results it returned.
Chunking strategies for RAG are the rules that decide where the text is split. Contracts, manuals, and support articles each need a different rule. This article compares 8 key RAG chunking strategies, explains what each one trades off, and shows how to choose the one that fits your use case. N-iX builds production RAG systems through its RAG development services and sets up retrieval testing alongside the architecture.
Why chunking decides what your RAG system can answer
The AI model never reads your full document library. It reads only the few chunks the search step passes to it, so anything the cuts left out is lost to it. A chunking choice shapes the answer a user receives at three stages:
- Search. A chunk that mixes two topics matches neither well, so the right passage can rank too low to be picked.
- Context. A chunk cut mid-argument reaches the model without its conclusion, or without the table header that gives the numbers meaning.
- Response. A chunk padded with unrelated text gives the model more material to misread.
The effect is measurable. Chroma’s research found that the choice of chunking strategy changed the share of relevant text the search found by up to 9%. Anthropic showed what lost context costs. Adding a short note on the source of each chunk before indexing cut the misses by 35%, from about 6 to about 4 in every 100 relevant passages. Adding the same notes to the keyword index brought the figure to about 3.
Related reading: RAG evaluation: In-depth guide to building reliable systems
When do you need chunking? 4 signs
Chunking is not always required. Anthropic notes that a knowledge base under 200,000 tokens, about 500 pages, can go directly into the prompt, with no need for RAG. If your collection is small and stable, test that option first. You need chunking when one or more of these signs apply:
- Your collection is larger than the model’s prompt. A knowledge base of thousands of pages won't fit in a single request, so the system has to select passages.
- Each question needs only a small part of the collection. Chroma observes that putting an entire collection into the context window often wastes processing and can distract the model. Context engineering decides what reaches the model.
- You handle many questions a day. Sending the whole collection with every question multiplies cost and response time. Prompt caching can lower that cost, so compare both approaches on your own questions.
- Answers must point to their source. Chunks map to specific passages or pages, so users and auditors can check where an answer came from.
8 key chunking strategies for RAG
We selected these eight RAG chunking strategies because each one solves a distinct issue, from cutting text cleanly to keeping context attached to each chunk. The first four apply fixed rules to your documents. The last four use an AI model while the documents are prepared, which adds cost and time before the system launches.
1. Fixed-size chunking
The system cuts text into pieces of equal length, counted in tokens (a token is roughly three-quarters of a word) or characters. A small overlap repeats the end of each piece at the start of the next, so ideas at the cut are not lost. It needs no AI model, so preparing documents involves no model calls, and runtime grows with the amount of text. The cuts ignore the text, so they can land mid-sentence or mid-table.
Best fit: uniform text with consistent formatting, and a first baseline for any project.
2. Recursive chunking
The system tries the largest natural break first, which is a paragraph. If the paragraph is too long for the size limit, it splits at line breaks, then at sentence ends, then at words. Short paragraphs stay whole, so chunks usually hold complete thoughts. It needs no AI model. It reads only text breaks, so it doesn't know where headings or tables begin.
Best fit: general text such as articles, policies, and support content. It is a sensible default to test first.
3. Document-structure chunking
The system follows the layout the author already built: pages, sections, or headings. A page or section is a unit someone chose on purpose, so these chunks often hold a coherent topic. Answers can also point to the exact page they came from, which helps users check them. For PDFs, the system first needs a step that reads each file’s layout.
Best fit: reports, filings, and other PDFs with meaningful pages or sections.
4. Parent-child chunking
The system stores each document at two sizes. Search matches the small chunks (the children), keeping matches precise. When a child matches, the model receives the larger surrounding text (the parent), such as the full section. This gives the model enough context to answer a broad question.
Best fit: knowledge bases where users ask both quick factual questions and broad analytical ones.
5. Semantic chunking
An embedding model turns each sentence into numbers that capture its meaning. The system then splits chunks where those numbers shift, usually marking a topic change. Chunk length varies with the content, so many tools let you cap it. In Chroma’s tests, capping chunk length improved results. The method adds an extra AI step to document preparation.
Best fit: documents that mix many topics, after simpler methods have been tested on your data and did not reach your accuracy target.
6. Contextual chunking
Before indexing, an AI model writes a short note for each chunk based on its source document. A chunk that reads “revenue grew 3%” then also carries information about the quarter when that growth happened. The note is attached to the chunk before indexing, so the search can match on it. Every chunk needs one AI call, so the cost grows with the number of chunks.
Best fit: documents whose chunks lose meaning on their own, such as financial reports and technical papers.
7. LLM-based chunking
A language model reads the document and decides where to cut based on its understanding of the content. It can follow structure that fixed rules miss. In Chroma’s working version, it first splits the text into small pieces, then the model picks the cut points between them. Each document needs a model call, so time and cost grow with the size of the collection.
Best fit: small collections of high-value documents, such as contracts or specifications.
8. Late chunking
Standard chunking cuts the text first and then turns each piece into searchable numbers on its own. Late chunking reverses the order. An embedding model that reads long inputs processes the whole document first, then the system cuts the result into chunks. Each chunk then carries signals from the surrounding text. A chunk that mentions only “the city” still carries the fact that the document is about Berlin. It needs no extra training and no AI call per chunk.
Best fit: long documents where later sections depend on earlier ones, such as contracts, manuals, and research papers, when your search platform supports a long-input embedding model.
RAG chunking strategies comparison
|
Strategy |
Advantages |
Drawbacks |
|
1. Fixed-size |
It has only two settings: chunk size and overlap. It needs no AI model, so it costs little to prepare and finishes quickly. Equal chunk lengths mean storage and cost follow directly from the volume of text. |
Related facts can end up in different chunks. Chunk size and overlap need tuning for each document type. |
|
2. Recursive |
It cuts at paragraph, line, and sentence breaks first, so chunks usually hold complete thoughts. It works on most plain text without special setup. |
It sees only the breaks in the text, so it has no view of page layout. Tables can be split in two. A heading may land in a different chunk from the text it introduces. With default settings, chunks can come out too short to carry context. |
|
3. Document structure |
It uses boundaries the author chose, so chunks often match real topics. Headings stay with their content. |
It needs a step that reads each file’s layout, which adds setup. It only works when files have a meaningful structure. A page can cut a topic in half, and a long section puts several topics into one chunk. |
|
4. Parent-child |
Small chunks keep the search precise. The larger parent gives the model enough context for a broad answer. One setup covers both short factual questions and long analytical ones. |
It needs a second store for the parent documents. Each question sends more text to the model, which raises running costs. A parent can carry unrelated text along with the match. |
|
5. Semantic |
Cuts follow topic changes, so each chunk covers one subject. It suits long documents that change topic without clear headings. |
It adds an AI step to document preparation. Chunk sizes vary, and very short chunks can lose context. Results depend on the sensitivity setting and on the embedding model. |
|
6. Contextual |
Each chunk carries its source, such as the company, report, or period, so it stays clear on its own. This helps the search tell apart chunks that look alike, such as the same table in different years’ reports. It can be added on top of any other chunking method. |
Every chunk needs an AI call, so cost grows with the size of the collection. Notes must be rewritten when a document changes. A wrong note can mislead the search, so spot-checks are needed. |
|
7. LLM-based |
The model reads the content like a person, so cuts can follow meaning that fixed rules miss. It can handle messy or mixed documents. |
It is the slowest and most costly option to prepare. Output can vary between runs, which makes results harder to reproduce. The model can alter the text when it copies it, so chunks need checking. |
|
8. Late chunking |
Each chunk retains context from the whole document, so references like “the company” or “this clause” keep their meaning. It needs no AI call per chunk. It works with any cutting rule, including simple ones. |
It needs an embedding model that can read long inputs, which limits tool choices. Documents longer than the model’s limit need extra handling. Long inputs use more memory and time. The benefit is smaller for short documents or for sections that stand alone. |
How to choose the best chunking strategy for RAG
The best chunking strategy for RAG is the one that scores highest on a company’s own documents and questions, within its accuracy and cost limits. No single RAG chunking strategy wins in every case, so the choice comes from testing. The work splits between two roles. The business owner sets the conditions. The tech partner turns them into tests and a recommendation.
3 factors that decide the best chunking strategy
- The type of document. A contract is built from numbered clauses, a support article from short paragraphs, and a financial report from tables. A rule that keeps a clause whole can split a table in half, so the same strategy performs differently on each.
- The type of question. “What is the notice period?” needs one short passage. “Summarize the termination terms” needs several paragraphs. Smaller chunks suit the first question, and larger chunks suit the second.
- The search technology behind the system. Every RAG system uses a component, called an embedding model, that decides which chunks look relevant. A strategy that ranks first with one model can rank lower with another. A result from a demo or a past project therefore needs to be rechecked after any technology change.
4 decisions for the business owner
- The use case and the questions that matter most. A support assistant, a contract library, and a financial reporting tool each need a different strategy. The use case shapes every later choice.
- The accuracy bar. The business owner defines what a wrong answer costs. A weak answer in internal search costs little. A wrong clause in a contract review costs a lot.
- Cost and delay limits. Some strategies call an AI model for every chunk, and others add setup time. The business owner sets a ceiling for preparation cost, running cost, and response time.
- Documents and a subject expert. The tech partner needs a representative set of documents and someone who can review a sample of real answers.
How the tech partner decides
The tech partner turns these conditions into benchmarks before the pilot begins. It then tests candidate strategies on the company’s documents, changes one setting at a time, and scores the search and the answer separately. Running cost is reported next to quality. The recommendation is based on documented results. The business owner approves it or asks for another round.
How N-iX approaches chunking strategies for RAG: 7 key practices
N-iX is a global technology partner for Pragmatic AI Software Engineering, and every engagement starts with evidence. For chunking, that means measuring each strategy's performance on your documents, infrastructure, and workflows before approving anything for a larger rollout. These are the seven key practices behind that approach.
- We change only the chunking and keep everything else fixed. During a comparison, the embedding model, search, ranking step, and prompt stay the same. Any difference in results then comes from the chunking.
- We build the test set from your documents and questions. It reflects real user behavior, your domain terms, edge cases, and questions that need several documents to answer. It includes factual and analytical questions, since they can favor different chunk sizes.
- We score the search and the answer separately. Context relevance shows whether the search found the right material. Faithfulness shows whether the answer stays within that material. Answer relevance shows whether the answer addresses the question. A chunking choice can improve one score and hurt another, and three scores show it.
- We pair AI scoring with expert review. Automated judges allow testing at scale. Subject-matter experts review sample answers and recalibrate the judges regularly to limit bias.
- We report cost next to quality and scale only on documented results. Latency and token use are measured for each chunking strategy, so you see what a gain costs before you pay for it. We move forward with another strategy only when the benchmarks show a gain.
- We test security and robustness before release. The tests cover prompt injection, sensitive data leaks, misleading passages, and questions the system should decline. Chunking matters here, because a chunk can carry unrelated or sensitive text along with the passage that matched.
- We monitor after launch. Retrieval quality, answer grounding, latency, token use, and drift are tracked through CI/CD and AI observability. Changes in your documents or models then show up early.
The right chunking strategy is the one that your own benchmarks support, and we put those benchmarks in place before the pilot starts.
FAQ
What is chunking in RAG?
Chunking cuts your documents into small pieces before a RAG system stores them. When someone asks a question, the system searches these chunks and passes the best matches to the AI model. The cuts decide which facts stay together, so a poor cut can separate a statement from the context that gives it meaning. N-iX, an experienced RAG development company, defines evaluation benchmarks alongside the architecture, so it measures chunking from the start.
What are the best chunking strategies for RAG?
No single strategy is best for every document set. The right choice depends on your document types, the questions people ask, and the cost you can accept. Start with a simple baseline such as recursive chunking, then test other strategies only where the results show a gap. Document-structure, parent-child, and contextual chunking are the usual candidates. N-iX compares these options against measurable results on your own documents before launch.
How big should chunks be, and how much should they overlap?
No universal setting exists. Short factual questions work better with smaller chunks, while broad analytical questions need larger ones. Overlap repeats text at each cut so that ideas are not lost. It also adds storage and can return the same text twice. The right balance depends on your documents and your search setup. N-iX tests size and overlap against your own questions.
Do long-context AI models make chunking unnecessary?
Only for small collections. If your whole knowledge base fits in the model’s prompt, you may not need RAG. Larger collections still do. Sending everything with every question raises cost and can distract the model from the relevant passage. N-iX helps teams choose the right approach for their documents and builds the RAG system when the collection outgrows the prompt.
How do we know a chunking strategy works for our documents?
Test it on questions your users actually ask and compare each strategy with a simple baseline. Score the search and the final answer separately, have an expert review a sample of real answers, and put the running cost next to the scores. N-iX builds this evaluation into its RAG development work from the start, so the choice rests on measured results.
Have a question?
Speak to an expert
