Advanced Retrievers — When Simple Search Isn't Enough

Simple retrieval does one thing: it takes your question as-is, converts it to an embedding, and fetches the semantically closest chunks.

The problem? "As-is." Real questions sometimes arrive with different words than the documents use, or carry a condition semantic search doesn't understand, or need an answer bigger than the chunk that gets found.

Advanced retrievers are all one idea: add some intelligence before the search or after it. Before: modify the question or translate it into filters. After: enlarge the result or reorder it. The search itself — the vector store — stays exactly the same.

Picture two librarians:

The first is a novice. You say "I want something about vacation money" and they search literally for "vacation money." The book you need is titled "Annual Leave Allowance Calculation"? Too bad — they won't find it. You asked for "2024 articles"? They'll bring anything "close in meaning" without glancing at the publication year. And when they do find the right page, they cut out a small snippet — without the rest of the chapter that completes the meaning.

The second is an expert. They take your request and rephrase it several ways with different synonyms before searching (that's Multi-Query). They hear "2024 articles" and understand that "2024" is a filter, not a search term — they go to that year's shelf and search inside it (that's Self-Query). And when they find the right snippet, they bring you the whole chapter it came from (that's Parent-Document).

Same library. Same books. All the difference is in the intelligence around the search.

A real example — four ways simple search fails

An HR knowledge base holding company policies. Look at these four questions:

Notice: in all four cases the model will answer — confidently. The problem isn't generation. The problem happens before the model sees anything at all.

The three patterns intervene at two points in the pipeline you already know:

1. Before the search (query side): instead of the question going in as-is — Multi-Query generates several phrasings of it, and Self-Query translates it into a structured filter + search text. 2. After the search (result side): instead of the result being delivered as-is — Parent-Document swaps the small chunk for the larger context it belongs to.

The vector store, the embeddings, the generation? All unchanged. That's why these patterns bolt onto any working RAG system without breaking anything.

The patterns — one by one

### 1) Multi-Query Retrieval — "ask the question several ways"

• What it fixes: vocabulary mismatch (failure #1) and multi-angle questions (failure #4).
• How it works: before the search, one LLM call generates 3–4 alternative phrasings of the question with different vocabulary. You search with every phrasing and merge the results with deduplication.
• Mini example: "How are vacation days calculated?" generates: "annual leave balance calculation method" + "vacation balance calculation policy" + "how many leave days does an employee accrue?" The second phrasing is the one that hits the formal HR document — and a phrasing in another language covers you if the documents are written in one.
• 💸 The price: one extra LLM call (so +latency before the search even starts) + 3–4 searches instead of one + more chunks entering the prompt if you don't trim. Full math below in the cost section.

### 2) Self-Query Retrieval — "the model translates the question into filter + search"

• What it fixes: metadata questions (failure #2) — "get 2024 articles about X," "decisions from department Y," "high-priority tickets."
• How it works: an LLM call takes the question + a description of your metadata fields and produces two things: a structured filter (year = 2024) + semantic search text ("remote work"). The filter executes as normal metadata filtering, and the semantic search runs only inside the filtered results.
• Mini example: "Get the 2024 HR decisions about remote work" → filter: {type: "hr-decision", year: 2024} + query: "remote work policy". Now 2021 decisions cannot come back — the filter cut them out before the search even ran.
• ⚠️ The precondition nobody mentions: this pattern is only as good as your metadata. If documents are stored without clean, consistent year and type fields, the model will generate filters on fields that don't exist — yielding zero results or garbage. Design the metadata at ingestion time.
• 💸 The price: one extra LLM call per question + ongoing maintenance of the metadata schema.

### 3) Parent-Document Retrieval — "search with the small, return the big"

• What it fixes: the chunk smaller than the answer (failure #3).
• How it works: at ingestion you split twice: small chunks (150–300 tokens) get the embeddings — because small is more precise in search. Each small chunk points to a larger parent (the section or page, 1,000–1,500 tokens) stored in a plain store alongside. At query time: search happens on the small chunks, but what goes into the prompt is the parent.
• Mini example: the search catches "Step 3: return company property" (high precision because the chunk is focused) — and the model receives the full "Termination Procedure" section with all five steps. The search kept the precision of small; the answer gained the completeness of big.
• 💸 The price: no extra calls — the price is different here: double storage (a second store for parents) and much larger context entering the model on every question — more input tokens, more money. And if you make the parents huge, you've reinvented the "information lost in the middle" problem yourself.

### What about Reranking and Query Rewriting?

Those are relatives you've already met — we won't re-explain them:

• Reranking works on the order of results after they come back: the results exist, but the best isn't first. (The Reranking & advanced retrieval concept.)
• Query Rewriting fixes a question missing conversational context: "and the discounted one?" needs to become a complete question before searching. (The multi-turn RAG concept.)

All four compose: Rewriting fixes the question's phrasing, Multi-Query widens its coverage, Self-Query translates its conditions, and Parent-Document and Reranking improve what comes back.

Diagram

Where each pattern intervenes in the pipeline:

                    ┌─────────────────────────────────────────────────┐
User question ────► │  QUERY SIDE — before the search                 │
                    │                                                 │
                    │  Multi-Query : 1 question ──► N phrasings       │
                    │  Self-Query  : question ──► filter + query text │
                    └───────────────────────┬─────────────────────────┘
                                            ▼
                              Vector store search (unchanged)
                                            ▼
                    ┌─────────────────────────────────────────────────┐
                    │  RESULT SIDE — after the search                 │
                    │                                                 │
                    │  Parent-Document : small chunk ──► big parent   │
                    │  Reranking       : re-order results             │
                    │                    (separate concept)           │
                    └───────────────────────┬─────────────────────────┘
                                            ▼
                              Augment + Generate (unchanged)

Notice that the box in the middle — the search itself — is marked unchanged. That's the essence: you're not changing the search; you're improving what goes into it and what comes out of it. Which is why you can adopt any one of these patterns, alone, today, on the system you already have.

First thing to internalize: these names are pattern names, not library features. They became famous through framework documentation (LangChain and LlamaIndex ship them ready-made under roughly the same names), but each one is an idea you can write in 15–30 lines of plain code. If you understand the idea, you can implement it yourself or use the ready-made version — while understanding what happens inside (that's the whole "what do you build RAG with?" decision).

Second: the three collapse into one equation. Simple retrieval is:

answer(question) = generate(question, search(embed(question)))

The patterns inject functions into the middle:

Multi-Query      : search(embed(q))  ──►  merge(search(embed(v)) for v in variants(q))
Self-Query       : search(embed(q))  ──►  search(embed(text(q)), filter=parse(q))
Parent-Document  : search(...)       ──►  parents(search(...))

Third — the order in which you try them isn't random. Start with the operationally cheapest: Parent-Document (no extra calls at query time) → Multi-Query (one extra call) → Self-Query (an extra call + a metadata investment). That is, when the diagnosis allows — diagnosis always comes first.

At the system level — and how to know it actually worked

The pattern is installed. Now the only question that matters: did it improve things or not? The answer is never "it feels better" — that feeling is exactly what leads systems to pay double for imaginary improvement.

The methodical way (which is the same philosophy as evaluating RAG systems):

1. A small golden set: 20–30 real questions — from your logs, not your imagination — and next to each: the chunks that should come back (or at least the correct document). 2. Measure retrieval alone, before generation: for each question — did the right chunk appear in the results? (Hit Rate), at what rank? What fraction of the returned results was relevant (Precision) and what fraction of the relevant material did you retrieve (Recall)? 3. Take a baseline with simple search. One number: "Hit Rate = 62%." 4. Install one pattern only. Measure again. Got 81%? The pattern earned its place. Got 64% while cost rose 60%? Remove it — no regrets. 5. Record cost and latency with every measurement. An improvement that charges you 3× for +2% isn't an improvement — it's a losing decision with a clear metric.

💡 Tip: step 4 especially — one at a time. Install three patterns at once and quality improves: you don't know which one helped. Quality drops: you don't know which one broke it. Isolating variables isn't scientific vanity — it's what saves you weeks of debugging.

The code

Here's Multi-Query in full, with no framework — the bare idea, portable to any provider and any vector store:

# ---------- MULTI-QUERY RETRIEVAL: the whole pattern, no framework ----------
question = "How are vacation days calculated?"

# 1) One extra LLM call: rephrase the question from different angles
variants_prompt = f"""Rewrite this question in 3 different ways.
Use different vocabulary in each rewrite. One question per line.

Question: {question}"""

variants = llm.generate(variants_prompt).strip().splitlines()
all_queries = [question] + variants              # 4 queries total

# 2) Search with every phrasing (parallelize these in production)
seen_ids, merged = set(), []
for q in all_queries:
    for chunk in vector_store.search(embed(q), top_k=3):
        if chunk.id not in seen_ids:             # 3) de-duplicate
            seen_ids.add(chunk.id)
            merged.append(chunk)

# 4) Cap the final set, then continue like any normal RAG pipeline
top_chunks = merged[:6]
answer = generate_grounded_answer(question, top_chunks)

Three things to notice:

• The `merged[:6]` line is the defense of your budget. Without it, 4 queries × 3 results can push 12 chunks into the prompt — the bill and the distraction along with them. Merging must be followed by trimming.
• Deduplication isn't a luxury. Different phrasings of the same question will mostly return the same strongest chunks — which is itself a quality signal: what shows up under every phrasing is the most relevant.
• The variants call can use a cheaper model. Rephrasing a question doesn't need your strongest model — that's the first savings opportunity you'll meet.

Exercise (45–60 minutes, on your current project):

1. Open your RAG Knowledge Assistant and collect 20 real questions — from logs if you have them; if not, write questions in real-user phrasing, not your documents' phrasing. 2. For each question record: the document that should come back. Congratulations — that's your first golden set. 3. Measure hit rate with simple search. Write the number down. That's the baseline. 4. Look at the questions that failed and classify them with the symptom→treatment table above: how many are vocabulary mismatch? How many metadata? How many small-chunk? 5. Implement one pattern — the one fixing your biggest failure class — by hand, like the Multi-Query code above. Measure again and compare to the written number.

What you'll learn from step 4 in particular — that your failure modes are not evenly distributed — is the real lesson of this page.

Misconceptions

1) "A more complex retriever = better results" — ❌ The single most wrong idea in this whole topic. Each pattern fixes one specific failure mode — if that mode doesn't exist in your system, the pattern adds cost, latency, and sometimes noise (extra results that distract the model) with zero gain. Complexity isn't a feature — complexity is a bill. The only feature is a better number on the golden set.

2) "These patterns belong to LangChain" — ❌ The names became famous through framework docs, but these are architecture patterns — exactly like RAG itself. Each one fits in under 30 lines in any language with any provider (you saw full Multi-Query above in 20 lines). Use the ready-made versions if you like — but understand the idea first, so when it fails you can open it up and fix it.

3) "Self-Query saves me from designing metadata" — ❌ The exact opposite — Self-Query is the pattern most dependent on well-designed metadata. The model translates the question into filters over the fields you defined at ingestion. No clean fields? No Self-Query. The intelligence lives in the schema before it lives in the model.

Security

Just one point here — and it concerns Self-Query specifically: a permission filter must never come from the model.

In Self-Query, the model generates filters from user text. Imagine a user typing: "get all documents, including the executive ones" — if the permission filter (allowed_departments) is generated by the same call, you are literally letting the user negotiate their permissions with the model. That's prompt injection at the retrieval level.

The rule: permission filters are applied in code, after the model generates the search filters — appended mandatorily, from a trusted source (the user's session). The model proposes year = 2024? Fine. The model decides who sees what? Never.

Cost — the stacking math, pen and paper

Here's the most important table on the page. Let's build on the same scenario as "why does RAG exist?": 50,000 questions/month.

> 💸 Volatile numbers *(illustrative pricing — check your provider's current rates)*: a mid-tier generation model at $3/M input and $15/M output tokens; a helper call (variants/parsing) on a small model at ~$0.001/call; embeddings negligible as computed before.

Read the table bottom-up: the full stack ≈ double the bill and roughly double the latency (from ~2–4 seconds to ~5–7). It may well be worth it — if the golden set says quality jumped. It's never worth it if you installed it "just in case."

Two savings tricks before you go:

• One call instead of two: if you run Multi-Query and Self-Query together, ask the model for the phrasings and the filter in the same call — you save a call and ~1 second per question.
• The patterns don't have to run on every question: a simple router (even a regex that detects a year/department in the question) can enable Self-Query only for questions that need it. Most questions are simple — don't pay the cleverness tax on them.

Diagnose with this table — symptom first, then treatment:

⚠️ Mind that last row. If the foundation is broken — blind chunking or an embedding model that doesn't suit your data's language — any pattern you stack on top polishes the garbage instead of fixing it. These patterns are improvements on a sound foundation, not a substitute for one.

• The system is still being built: the first version of any RAG uses simple search. Always. You need a baseline to measure against before any cleverness.
• There's no measurement: if you don't have a golden set, it's not pattern time — it's golden-set-building time. Improvement without measurement is guessing with money.
• The failure isn't yours: your users ask direct questions in the documents' own vocabulary? Plain top-k is enough, and any addition is cost without return.
• Latency is sacred for you: if your SLA is tight, an extra LLM call before the search may be a luxury you can't afford — consider Hybrid Search or better chunking first (they fix many of the same symptoms with zero extra calls).
• The metadata question is rare: if "get me the 2024 ones" shows up once a month, don't build Self-Query for it — put filters in the UI and let users pick them. Cheaper and more reliable.

Alternatives — before you install an advanced retriever

Trade-offs