Agentic RAG: What It Fixes, What It Costs, and What It Still Cannot Do

TL;DR

Agentic RAG replaces the single retrieve-then-generate pass of classic RAG with an agent that plans its retrieval: rewriting queries, choosing sources, running multiple searches, checking whether the evidence actually answers the question, and retrying when it does not. It genuinely fixes the single-shot failure modes, and it costs more per query by design, since one question can trigger several model calls and searches. What it does not fix is the knowledge itself: an agent that retries against a corpus of stale or contradictory documents converges on stale answers with more confidence. This guide covers the architecture, the honest trade-offs, and the boundary where retrieval stops being the problem.

How is agentic RAG different from classic RAG?

Classic RAG is a fixed pipeline: embed the query, fetch the nearest chunks, paste them into the prompt, generate. Agentic RAG puts a decision-maker in the loop. The moves the agent adds:

1. Query planning. The question is decomposed or rewritten before retrieval, so "why did churn rise in Q2" becomes separate searches for churn data, Q2 changes and customer feedback rather than one embedding lookup.

2. Source routing. The agent chooses where to look per sub-question: the vector store, a SQL database, a knowledge graph, a web search, an API. Classic RAG has one retriever; agentic RAG has a toolbox.

3. Self-checking. After retrieval, the agent evaluates whether the evidence supports an answer, and retrieves again with a reformulated query when it does not, instead of generating from whatever came back.

4. Multi-hop assembly. Questions whose answers span documents get resolved stepwise, with each hop's result shaping the next query.

The through-line: retrieval becomes a loop with judgment instead of a lookup.

What does it measurably fix?

The failure modes of single-pass RAG that an agentic loop addresses:

1. Bad first retrieval is no longer fatal. Classic RAG's quality ceiling is set by the first nearest-neighbor lookup. A retry loop with query reformulation recovers a large share of these misses.

2. Multi-hop questions become answerable. "Which customers were affected by the outage the payments migration caused" requires joining facts that never co-occur in one chunk. Stepwise retrieval handles this; similarity search does not.

3. Heterogeneous sources compose. Structured data, documents and live APIs can each answer the part of the question they are good at.

4. Irrelevant context shrinks. Because the agent evaluates evidence before generating, less near-miss material reaches the final prompt, which also mitigates context rot, the measured decline in output quality as context grows.

What does it cost?

The trade-offs, stated plainly rather than buried:

1. Latency multiplies. One question can become three to ten model calls plus retrievals. Interactive use cases feel this immediately.

2. Spend multiplies with it. Every planning step, retrieval evaluation and retry is billed input and output. An agentic pipeline that averages five calls per question costs roughly five times its single-pass ancestor on the same model.

3. Failure gets harder to debug. A wrong answer can now originate in decomposition, routing, any hop's retrieval, or synthesis, which is why agent observability tooling has converged on full-trace capture.

4. Non-determinism compounds. Each judgment point is a place where identical inputs can produce different paths.

A reasonable rule from production deployments: route simple lookups through the cheap single-pass path and reserve the agentic loop for questions that fail it. Most traffic is simple; most value is in the hard tail.

What can agentic RAG still not do?

Here is the boundary that matters for anyone evaluating this architecture: agentic RAG improves the search, not the knowledge. Three failures survive any number of retries:

1. Staleness with confidence. If the corpus contains last quarter's policy alongside this quarter's, a diligent agent retrieves both, and nothing in either document says which is current. Recency sorting is a heuristic, not a guarantee, because the newest document is not always the authoritative one.

2. Contradictions between systems. When the CRM and the contract disagree, retrieval faithfully delivers both. Adjudicating which is true requires provenance and authority metadata that documents do not carry.

3. Facts that were never written down. The decision made in a meeting and reversed in a thread exists in no retrievable chunk unless something captured and resolved it at the time.

These are properties of a memory layer, not a retrieval strategy. A bi-temporal store like Sentra records when each fact became true and when it stopped being true, detects contradictions at write time, and serves the current governed fact to whichever agent asks. Agentic RAG then has a corpus worth searching. The two compose: the loop provides judgment at read time, the memory layer provides truth maintenance at write time.

FAQ

The decision rule

Adopt agentic RAG when your failing questions are multi-hop or multi-source, and pay its cost only on the traffic that needs it. If your failing answers are instead stale or contradictory, no retrieval loop fixes that; the corpus needs a truth layer before the searcher needs a brain.