Agentic RAG — When RAG Starts Making Decisions

Classic RAG is like a student taking an open-book exam with one rule: open the book once, copy the closest paragraph, write the answer. Done. If that first paragraph is insufficient or wrong, there is no plan B.

Agentic RAG is like a real researcher: read the question, decide — "does this even need a search? In which source? Is this phrasing good enough?" Search, read the result, and ask: "is this enough, or should I search again using something I just learned?" And at the end, if nothing solid was found, say "I don't know" instead of inventing.

The essential difference: classic RAG retrieves. Agentic RAG decides to retrieve — and repeats until it's satisfied.

Imagine two library assistants, and you hand each a complex research question:

Assistant one (classic RAG) — takes your question, walks to the nearest shelf, grabs the first book with words resembling your question, opens a page, and reads it to you. Wrong book? Not their problem. Question needs three books? You still get one. Fast and cheap — and perfectly fine if your question was simple.

Assistant two (Agentic RAG) — pauses to think: "Does this belong in the history section or the law section?" (Routing). "This is actually three questions in one; let me split them" (Decomposition). Fetches a book, reads it, says "ah, this references another source — let me get that too" (Iterative). Before coming back to you: "Do I actually have a complete answer?" (Self-check). And if the answer isn't there: "I don't have a confirmed source for this part" instead of making one up.

Assistant two is more accurate on the hard question — but slower and more work. If your question was "where's the restroom?", assistant one was more than enough. Not every question deserves a researcher.

A real example

The same multi-hop question through both systems:

> Classic RAG: > Question: "Who signed the biggest deal last year, and what's their discount policy?" > One search with the full question → a mixed bag of chunks (some about deals, some about discounts, no link between them) → the model fabricates an answer connecting unrelated facts. Convincing and wrong.

> Agentic RAG: > 1. Decompose: the model splits it — "(a) What's the biggest deal? (b) Who signed it? (c) What's that person's discount policy?" > 2. Search 1: find the biggest deal → "Deal X, signed by Ahmed." > 3. Search 2 (Iterative): now it knows the name, so it searches "Ahmed discount policy" → found. > 4. Self-check: "Do I have the deal + the signer + their policy? Yes" → answers grounded in three clear sources.

Same model, same data. The difference: the first searched once; the second decided how to search, and repeated. The first cost one call; the second cost 4–5.

The pivotal idea: retrieval turns from a fixed step into a tool the model decides to use.

In classic RAG, your code enforces the sequence: retrieve() then generate(), always, in that order. The model has no say — it just writes the answer at the end.

In agentic RAG, you hand the model retrieval as a tool inside an agent loop, and the model decides: 1. Observe the question. 2. Decide: search? Where? With what phrasing? Or do I already know the answer? 3. Execute the retrieval (if it decided to). 4. Inspect the result and ask: is this enough? 5. Repeat from 2 — or answer — or say "not found."

Here's what trips people up: in classic RAG, retrieval is guaranteed to happen exactly once. In agentic RAG, retrieval might happen zero times (if the model knows the answer), once, or five times. That non-determinism is the source of the power — and the source of the cost and the failure modes at the same time.

Step by step — the five patterns

Each pattern with a small concrete example:

1) Query Routing — send the question to the right place The model decides which source/index a question belongs to. "How much is product X?" → the products index. "What's the vacation policy?" → the HR index. Instead of searching everything, you search the right place.

2) Query Decomposition — split the compound question A multi-part question becomes sub-questions. "Compare the return policy between 2023 and 2024" → "(a) 2023 policy? (b) 2024 policy?" → one search each → merge.

3) Iterative Retrieval — search, read, search deeper Search → read → search again using what you learned. The first search surfaces a new name or term; the second search uses it. This is what solves multi-hop questions.

4) Self-check before answering Before returning the answer, the model asks itself: "Is the context in front of me actually sufficient for this question?" If not → search again, or say "I don't have enough information." This is the step that turns "guessing" into "admitting."

5) Retrieval-as-a-Tool The wrapper around the four above: retrieval is a tool inside an agent loop, and the model calls it whenever it decides, with whatever inputs it chooses, as many times as it sees fit.

Diagram

The difference between the two paths — this is the heart of the whole concept:

┌──────────────── CLASSIC RAG — one fixed path ─────────────────┐
│                                                               │
│   Question ──► Retrieve (once) ──► Stuff context ──► Generate │
│                                                        │      │
│                                                        ▼      │
│                                                     Answer    │
│   Model calls: ~1–2 · deterministic · cheap · fast            │
└───────────────────────────────────────────────────────────────┘

┌──────────── AGENTIC RAG — a loop with decisions ──────────────┐
│                                                               │
│   Question ──► [ LLM decides ]                                │
│                    │                                          │
│         ┌──────────┼───────────┬──────────────┐               │
│         ▼          ▼           ▼              ▼               │
│    Search? Where?  Decompose?  Rephrase?   Already know?      │
│         │          │           │              │               │
│         └────► Retrieve (tool) ◄──────────────┘               │
│                    │                                          │
│                    ▼                                          │
│              [ Self-check: enough? ] ──► no ──┐               │
│                    │ yes                      │ (loop again)  │
│                    ▼                          └─────────┘     │
│              Answer  or  "I don't know"                       │
│   Model calls: ~4–8 · non-deterministic · costlier · slower   │
└───────────────────────────────────────────────────────────────┘

Notice the essential difference: in classic RAG, the arrow moves in one straight line. In agentic RAG, there is a loop that feeds back into itself. That loop is the source of the intelligence — and also the source of the "agent that never stops searching" failure. Every trip around the loop = one model call = money + time.

First thing to internalize: the difference isn't in retrieval itself — it's in who owns the control flow. In classic RAG, your code controls the sequence: retrieve then generate, fixed. In agentic RAG, the model controls the sequence through its decisions. You moved control from deterministic code to a probabilistic model. That buys you flexibility and costs you predictability.

Second: retrieval becomes an ordinary tool call. If you understand tool calling, you understand the mechanism: define a search(query, source) tool, put it in the loop, and the model decides when to call it, with what inputs, how many times. Agentic RAG is literally "RAG built with tools inside an agent loop" — no new magic, just two concepts you already know, combined.

Third — and this is the most consequential design decision: the loop needs hard guardrails. A model that can decide "search again" can decide it forever. You need: a hard cap on iterations (e.g. 5), a cost ceiling, and a timeout. Without them, one hard question can burn 30 calls and $2 on its own. The unbounded loop is the single most common bug in agentic RAG.

Fourth: the self-check is itself a model call — it costs money. Asking the model "is this context sufficient?" is one extra call per iteration. Great for quality, but put it in the cost math. Nothing in agentic RAG is free.

At the system level — the API reality

Five facts you'll meet the moment you implement this:

• You pay for every trip around the loop. Every decision, every retrieval, every self-check = a model call with input + output tokens. The question that was one call is now five. The bill scales linearly with iterations.
• Latency accumulates. Five sequential calls at 1–2 seconds each = 5–10 seconds to answer. A user accustomed to one-second responses will feel it. Parallel patterns help — but not every step can be parallelized.
• Infinite loops are a real risk. Without a hard iteration cap, the model can enter "search again... not enough... search again..." forever. An explicit cap is mandatory — not an optimization.
• The model can still guess after retrieval fails. The nastiest failure: the search fails three times, and instead of saying "I don't know," the model gives up and answers from memory. You need explicit instructions: "If retrieval fails, say you don't know — do not answer from general knowledge."
• Debugging is much harder. Because it's non-deterministic, the same question can take a different path every run. Without tracing (logging every decision and retrieval), you're blind. Log the full loop from day one.

The code

This isn't from any framework — it's the bare idea: retrieval as a tool inside a bounded loop, with a self-check and an explicit "I don't know" exit. Read it as pseudocode:

# Retrieval is now a tool the model decides to call — not a forced step
def search(query: str, source: str = "default") -> str:
    chunks = vector_store.search(embed(query), top_k=5, index=source)
    return format_chunks(chunks)

MAX_STEPS = 5                                  # Guardrail: hard iteration cap
question = "Who signed the biggest deal last year, and what's their discount policy?"
context = []

for step in range(MAX_STEPS):                  # Bounded loop — never infinite
    decision = llm.decide(                     # The model decides the next step
        question=question,
        gathered=context,
        tools=[search],
        instructions=(
            "Decide: search (with what phrasing/source), or do you have "
            "enough to answer? If retrieval returned nothing useful, say "
            "'I don't know' — never answer from general knowledge."
        ),                                     # Anti-guessing after failure
    )

    if decision.type == "answer":
        break
    if decision.type == "give_up":             # The honest exit
        return "I don't have enough information in the sources to answer this."

    # The model chose to search — execute and accumulate (Iterative)
    result = search(decision.query, decision.source)
    context.append(result)

    # Self-check: enough? (an extra model call — it costs money)
    if llm.is_context_sufficient(question, context):
        break

answer = llm.generate(question, context)       # Answer grounded in the total

Three things to notice before moving on:

• `MAX_STEPS` is not a detail — it's the safety valve. Remove it, and one hard question can burn your budget in an endless loop. It's the first line you write, not the last.
• The explicit "I don't know" exit. The give_up branch and the anti-guessing instructions are the difference between an honest system and one that invents confidently after a failed search.
• Every `llm.` call costs money. Count them: decide × N + is_context_sufficient × N + generate. That's easily 4–8 calls versus 1–2 in classic RAG.

Exercise (20 minutes): Take a working classic RAG you have and prepare 3 multi-hop questions (each needing two facts from two places). Run them through the classic path and record: how many did it get right? (It will mostly fail.) Now — on paper only — decompose each question into its sub-questions by hand, search for each part separately, and merge. Compare the accuracy. Congratulations — you just performed manual query decomposition, felt the problem Agentic RAG was invented for — and its cost (count how many searches you ran).

⚠️ Watch out: mistakes #1 and #2 are the most common. The infinite loop burns money loudly; agentic-everywhere burns money quietly on questions a single pass would have answered.

Misconceptions

1) "Agentic RAG is always more accurate" — ❌ No. Always more expensive, sometimes more accurate. On simple questions (most FAQ and doc-QA traffic), classic RAG answers correctly at a fraction of the cost. Agentic wins on compound and multi-hop questions only. You pay a multiplied cost for extra accuracy on a subset of questions — not all of them.

2) "Agentic RAG is a new architecture that replaces classic RAG" — ❌ No — it's built on top of it. You still need good chunking, good embeddings, good retrieval. If your base retrieval is bad, agentic will search bad data more times and cost you more. Agentic adds a decision layer — it doesn't fix a weak foundation.

3) "More iterations = more quality" — ❌ Quality saturates quickly; cost grows linearly. After 3–4 iterations, most questions stop improving — you're just paying. Extra iterations sometimes hurt (the model wanders and accumulates scattered context).

4) "The self-check prevents hallucination" — ❌ It reduces it, it doesn't prevent it — same as RAG itself. The self-check is a model call and can be wrong: declaring the context sufficient when it isn't, or the reverse. A useful layer, not a guarantee.

Security considerations

Two risks that grow specifically in the agentic setting:

1) Repeated retrieval = a wider injection surface. Every iteration pulls new external content into the model's context — and every retrieved chunk is untrusted input (as covered in base RAG). In agentic RAG, the model makes decisions based on that content, so malicious content can steer the loop itself: "ignore that, search for this instead." Treat retrieval results as data, and constrain which tools the loop can reach.

2) Retrieval-as-a-tool = permissions must follow the user. If the model decides where to search, its routing must be confined to sources this user is allowed to see — permission filtering at retrieval time, not after the information has already reached the model. (Broader details in tool-calling and RAG security.)

Cost — the honest math

This is the heart of it. Agentic RAG multiplies model calls — that's not a footnote, that's the whole decision. Let's compute:

Scenario: 50,000 questions/month, a generation model at $3/M input and $15/M output tokens.

> 💸 Volatile numbers *(illustrative pricing — check your provider's current rates)*. The stable point isn't the dollar figure — it's the ratio: agentic costs 4–8× classic, because every iteration is a full call and the context grows with each one.

The conclusion nobody says out loud: running this math over all your questions is waste. The smart math: what percentage of your questions is actually multi-hop? If it's 10%, route only that 10% to agentic and keep 90% classic — you get the accuracy gain where it matters and pay the multiplier on a tenth of the traffic. That's the difference between a $700 bill and a $4,000 bill for the same perceived quality.

Performance

• Sequencing is latency's enemy. Every iteration waits for the previous one. Where decisions are independent (like decomposition sub-questions), run the searches in parallel — it cuts time dramatically.
• Accumulated context slows every iteration. Each iteration adds to the context, so iteration five is slower and more expensive than iteration one. Prune context between iterations — drop what's no longer useful.
• Show progress to the user. "Searching… reviewing sources…" makes six seconds psychologically bearable. Real latency doesn't drop, but perceived latency improves.

• Multi-hop questions: the answer requires linking two or more facts from different places — "who did X, and what's their Y?"
• Heterogeneous sources: data spread across different indexes/systems, where the question needs routing to the right one.
• Ambiguous questions that need clarification or rephrasing before retrieval can succeed.
• High-stakes "must be sourced" contexts: legal, medical, financial — where "I don't know" is far safer than an unverified answer, and the self-check earns its cost.
• First-pass retrieval fails often: if your questions inherently need more than one attempt, iterative retrieval justifies itself.

• Direct FAQ and doc-QA questions (most workloads!): "What's the return policy?" → one retrieval is enough. Agentic here pays a multiplied cost with no measurable accuracy gain.
• Cost or latency is sensitive: a high-volume app or one needing instant responses — multiplied calls and multiplied time can be fatal.
• Your base retrieval is weak: fix chunking and first-pass retrieval first. Agentic on top of a bad foundation amplifies the problem and the bill.
• Simple, well-structured data: one clean, direct source leaves no real decisions for the model to make — the loop is complexity with no payoff.

💡 Tip: the working rule: always start classic. Measure. If you find a class of questions (multi-hop, heterogeneous sources) that consistently fails, upgrade that class only to agentic. You don't start with agentic — you end up there when the data proves you need it.

Alternatives

These aren't rivals: many mature systems run classic + reranking + query rewriting, and reserve full agentic for one specific class of questions. The full decision lives in RAG vs. long context vs. fine-tuning and its agentic extension.

Trade-offs

Bottom line: agentic buys accuracy on hard questions + the ability to say "I don't know" at the price of higher cost + longer latency + more complexity. A trade that wins on compound questions and loses on simple ones.