Embeddings & Vector Search· Lesson 4 of 7
Chunking — Splitting Your Documents
In one line · Before embedding a long document, you split it into smaller chunks so each piece carries one focused meaning that's easy to retrieve.
Bad chunking is the top reason RAG returns weak results: too-big chunks blur the meaning, too-small ones cut the context and lose it.
Like breaking a book into paragraphs instead of one block: each paragraph is one idea, so a search lands on the exact paragraph, not the whole book.
Like cutting a long lecture into study cards, one idea per card: at exam time you pull just the card you need, not the whole notebook. But if you cut a sentence in half, the card comes out incomplete and useless.
A 50-page manual split into ~300-word chunks with slight overlap; a query on 'refund policy' returns just the chunk with the policy, not the entire chapter. The overlap keeps a key sentence from getting sliced mid-thought.
You pick a chunk size (in tokens or words) and an overlap between consecutive chunks. Size drives a tradeoff: small chunks = focused meaning and precise matching, but less context for the model; large chunks = full context, but the vector represents a blend of ideas so matching gets fuzzy. Overlap (typically 10–20%) guarantees a sentence at a boundary appears whole in at least one chunk. Smarter than blind fixed-size splitting: cut at natural boundaries — paragraphs, headings, sections — so each chunk stays one coherent idea.
🧪 Try it: take a long document and chunk it 3 ways: ~100-word, ~300-word, and ~1000-word chunks (same overlap). Embed all three and run the same 5 queries against each version. You'll see it firsthand: small chunks match precisely but arrive context-starved, big ones carry full context but match mushily — and the middle size usually wins.
The biggest mistake: blind fixed-character splitting that slices sentences and tables mid-way — cut at paragraph and heading boundaries instead. Second: huge chunks 'for context' — their vectors turn muddy and match no specific query well. Third: forgetting to store metadata (document name, section number) with each chunk, so when it's retrieved you can't attribute the source or fetch its neighbors.
You must chunk whenever documents run longer than a few paragraphs: manuals, contracts, internal docs, articles — practically every real RAG system. And when content has heading structure, exploit it for the splits.
If your documents are already short and focused — tweets, FAQ entries, support tickets — leave them whole; chunking would break healthy units of meaning. And if questions always need the full document (contract summarization, say), return the whole document, not a slice.