Agentic RAG: Build Retrieval Loops That Decide

Agentic RAG is not "RAG with an agent somewhere." It is a retrieval architecture in which the system decides whether to retrieve, what to retrieve, whether the evidence is sufficient, and when to stop. A traditional pipeline sends every user request through the same path: embed the query, fetch the top ten chunks, and generate an answer. That works for simple lookup questions, but it wastes tokens on greetings, misses multi-part questions, and cannot safely call a database or internal API.

This guide shows how to design the decision loop itself. It covers when agentic retrieval is worth the complexity, what state the loop needs, how to rewrite and decompose queries, how to use tools without losing access control, how to judge evidence sufficiency, and how to evaluate decision quality rather than only answer quality.

The companion ideas are simple: retrieval is a tool, the model is a planner over that tool, and every loop needs a stop condition. If you remove any of the three, you usually have either a fixed RAG pipeline or an expensive agent that cannot prove why it retrieved something.

Manual link review: see Related OurToken Guides below.

Agentic RAG vs Traditional RAG

A traditional RAG pipeline is deterministic: every request follows one retrieval path and one generation path. Agentic RAG adds conditional control. The controller can be a finite state machine, a graph, a rule engine, or a model-assisted planner, but the important part is not the framework. It is that different requests take different paths for an observable reason.

The distinction matters because the two systems have different failure modes. Traditional RAG often fails quietly: the retriever returns plausible-looking chunks, and the model writes a fluent answer from incomplete evidence. Agentic RAG fails loudly when designed well: the trace shows that the query was rewritten, which tool ran, which evidence was accepted, why another iteration started, and why the loop stopped.

The Four Decisions That Make RAG Agentic

Before choosing a framework, define these decisions:

1. Retrieve or not. A greeting, identity question, or harmless small talk usually should not hit the index.
2. How to retrieve. The system may search one vector collection, rewrite the query, decompose a multi-hop question, call a SQL reader, or query a support API.
3. Is the evidence sufficient? The loop needs a typed judgment, not a vague feeling that the context looks relevant.
4. Stop or continue. Every additional iteration must reduce uncertainty enough to justify latency and cost.

If your application cannot describe those four decisions in a trace, it is not yet an agentic RAG system. It is a traditional pipeline with extra model calls.

Decide When an Agentic RAG Pipeline Is Worth It

Agentic retrieval adds latency, state, testing, and security work. It is usually worth it when the answer depends on more than one source, the query intent varies widely, or a deterministic retrieval path cannot know the required scope in advance.

Start with one workflow that already frustrates a fixed pipeline. A customer-support assistant that must read tickets, invoices, and runbooks is a good first target. A small FAQ bot usually is not.

Design the Loop State Before Writing Prompts

The agentic RAG architecture needs a shared state object. Without it, model calls become conversation soup and the trace cannot explain why the system acted.

from dataclasses import dataclass, field
from datetime import datetime
from enum import Enum


class NextAction(str, Enum):
    ANSWER = "answer"
    RETRIEVE = "retrieve"
    USE_TOOL = "use_tool"
    REWRITE = "rewrite"
    DECOMPOSE = "decompose"
    ASK_HUMAN = "ask_human"
    REFUSE = "refuse"


@dataclass
class Evidence:
    evidence_id: str
    source_id: str
    source_type: str
    text: str
    score: float
    retrieved_at: str
    query_version: int


@dataclass
class RetrievalState:
    request_id: str
    owner_id: str
    original_query: str
    working_query: str = ""
    subqueries: list[str] = field(default_factory=list)
    evidence: list[Evidence] = field(default_factory=list)
    tool_trace: list[dict] = field(default_factory=list)
    iterations: int = 0
    max_iterations: int = 3
    next_action: NextAction = NextAction.RETRIEVE
    missing_information: list[str] = field(default_factory=list)

    def seen_source_text(self, text: str) -> bool:
        normalized = " ".join(text.lower().split())
        return any(
            " ".join(item.text.lower().split()) == normalized
            for item in self.evidence
        )

The state should contain only what the loop needs to make a decision. Do not put entire database rows into state. Store projections and evidence IDs, then keep the full source records in your application database.

A production trace should record at least:

request_id
owner_or_tenant_id
original_query
working_query
subqueries
selected_tools
retrieved_source_ids
evidence_scores
sufficiency_decision
missing_information
iteration_count
model_routes
latency_ms
token_usage
stop_reason

This trace is what separates an agentic system from a guess. When an answer is wrong, you should be able to tell whether retrieval missed the evidence, the planner chose the wrong tool, the sufficiency check accepted weak context, or generation ignored valid evidence.

Classify the Request Before Retrieving

The first decision is often the cheapest: should the system retrieve at all? A deterministic classifier can handle obvious cases, while a small model route can classify ambiguous requests. Keep both behind one function so policy remains visible.

def classify_query(query: str) -> NextAction:
    normalized = query.strip().lower()

    if len(normalized) < 4:
        return NextAction.ANSWER

    if any(normalized.startswith(word) for word in {"hi", "hello", "thanks"}):
        return NextAction.ANSWER

    if "invoice" in normalized and "why" in normalized:
        return NextAction.USE_TOOL

    if " and " in normalized or "compare" in normalized:
        return NextAction.DECOMPOSE

    return NextAction.RETRIEVE

This is adaptive retrieval RAG: the system chooses whether, where, and how to search per request. Production classifiers usually combine rules, embeddings, and a model-assisted route. But even a crude router shows the value: the system stops paying retrieval tax for requests that never needed evidence.

For model-assisted routing, ask for structured output:

Classify the user request for a retrieval system.

Return JSON with:
- action: answer | retrieve | use_tool | decompose | ask_human
- reason: one short phrase
- missing_information: array of concrete facts needed

Rules:
- Do not retrieve for greetings or pure identity questions.
- Use a tool only if the named tool is listed as available.
- Use decompose only for questions requiring multiple independent facts.
- Do not include private data in the reason.

Then validate the JSON before trusting the action. An invalid classifier response should fall back to a safe default, not trigger an arbitrary tool call.

Rewrite and Decompose Queries When Needed

User language is often different from document language. "Customer cannot pay" may need "payment failure," "declined transaction," and "billing error." Agentic retrieval can rewrite the query, but the rewrite must preserve intent and produce auditable changes.

def rewrite_query(state: RetrievalState, model_output: dict) -> RetrievalState:
    if model_output.get("action") not in {"retrieve", "use_tool"}:
        return state

    state.working_query = model_output.get("rewritten_query", state.original_query)
    state.subqueries = model_output.get("subqueries", [])[:3]
    state.missing_information = model_output.get("missing_information", [])
    return state

The model prompt should return a bounded plan:

You are improving a retrieval query.

Return JSON:
{
  "action": "retrieve | use_tool",
  "rewritten_query": "...",
  "subqueries": ["...", "..."],
  "missing_information": ["...", "..."]
}

Rules:
- Preserve the user's intent and tenant scope.
- Do not add filters that were not requested.
- Maximum two subqueries.
- Do not invent IDs, dates, product names, or customer names.
- Prefer domain vocabulary over synonyms.

Use decomposition only when subqueries are independent enough to be useful. "Compare the 2025 and 2026 SLA" can become two policy lookups. "Why is the site slow?" often needs one incident lookup, one deployment lookup, and one status lookup, but you should not decompose blindly into ten fragments.

Keep Query Rewrites Reversible

RAG query rewriting must always retain the original query. If a rewritten query retrieves the wrong tenant scope, drops a date, or turns a broad request into an unrelated product name, the original query is needed for retry and debugging.

original_query: customer cannot pay invoice 88123
working_query: payment failure invoice 88123
subqueries: ["invoice 88123 payment status", "invoice payment failure codes"]

The rewritten query is a retrieval aid, not the source of truth.

Use Tools as First-Class Retrieval Sources

Vector search is excellent for prose and weak for exact state. A question such as "why did payment 4471 fail?" may need the payment service, not the nearest policy paragraph. In agentic RAG, tools participate in retrieval, but they must obey the same authorization rules as any other API.

from dataclasses import dataclass
from typing import Callable


@dataclass
class RetrievalTool:
    name: str
    description: str
    allowed_roles: set[str]
    handler: Callable[..., list[Evidence]]


def search_runbooks(query: str, owner_id: str) -> list[Evidence]:
    return vector_search(
        collection="runbooks",
        query=query,
        owner_id=owner_id,
        top_k=5,
    )


def read_invoice_failure(invoice_id: str, owner_id: str) -> list[Evidence]:
    invoice = invoice_api.get_projection(
        invoice_id=invoice_id,
        owner_id=owner_id,
        fields=["invoice_id", "status", "failure_code", "updated_at"],
    )
    return [
        Evidence(
            evidence_id=f"invoice-{invoice['invoice_id']}",
            source_id=invoice["invoice_id"],
            source_type="invoice_api",
            text=invoice_to_text(invoice),
            score=1.0,
            retrieved_at=datetime.utcnow().isoformat(),
            query_version=1,
        )
    ]


TOOLS = {
    "search_runbooks": RetrievalTool(
        name="search_runbooks",
        description="Search internal runbooks and policy documents.",
        allowed_roles={"support_agent", "admin"},
        handler=search_runbooks,
    ),
    "read_invoice_failure": RetrievalTool(
        name="read_invoice_failure",
        description="Read the current failure projection for one invoice.",
        allowed_roles={"support_agent"},
        handler=read_invoice_failure,
    ),
}

The tool layer should enforce:

• Tenant ownership on every lookup.
• Role-based access.
• Argument validation.
• Read-only scope unless the tool is explicitly a mutation.
• Rate limits and timeouts.
• A projection of fields, not the full database object.

This is where agentic RAG overlaps with agent guardrails. The AI agent guardrails guide shows how to place input, tool, and output checks around the loop. A retrieval planner should never be allowed to invent authority; it can propose a tool, but policy decides whether the tool runs.

The Tool Contract Matters More Than the Tool Name

Each retrieval tool should document:

name
purpose
required_arguments
optional_arguments
owner_scope
allowed_roles
return_fields
freshness
failure_modes
cost_estimate

Without this contract, the model will choose tools from their names alone. That works in a demo and fails in production.

Judge Evidence Sufficiency Explicitly

"I found five chunks" is not evidence sufficiency. The loop needs to know whether the retrieved facts answer the current subquestion, conflict with one another, or merely share vocabulary with the query.

A useful evidence judgment contains:

{
  "action": "answer",
  "confidence": 0.82,
  "reason": "Payment status and failure code are present.",
  "answered_subqueries": ["invoice 88123 payment status"],
  "missing_information": ["refund eligibility"],
  "conflicts": []
}

The judge can be a model-assisted call wrapped in deterministic validation. A minimal contract keeps the loop testable:

def judge_sufficiency(state: RetrievalState, evidence: list[Evidence]) -> dict:
    """Return a typed judgment for the controller.

    Required keys: action, confidence, reason, answered_subqueries,
    missing_information, conflicts.
    """

The controller should combine this with deterministic checks:

Do not rely only on embedding similarity. A document about "payment failures" can rank highly even when it does not mention the specific invoice. Store source type, entity IDs, timestamps, and access scope so the judge can distinguish relevance from authority.

Control the Context Budget

Agentic loops can accumulate evidence quickly. Deduplicate before each generation step:

def deduplicate_evidence(state: RetrievalState, max_items: int = 8) -> list[Evidence]:
    selected = []
    seen_texts: set[str] = set()
    for item in sorted(
        state.evidence,
        key=lambda evidence: evidence.score,
        reverse=True,
    ):
        normalized = " ".join(item.text.lower().split())
        if normalized in seen_texts:
            continue
        seen_texts.add(normalized)
        selected.append(item)
        if len(selected) >= max_items:
            break
    return selected

Then assemble the prompt with source labels:

Evidence:
[invoice-88123] Status: failed. Failure code: card_declined. Updated at 2026-09-09.
[runbook-pay-014] For card_declined, ask the customer to update the payment method.
[sla-2026] Support response target is four business hours for Priority 2.

The model should cite evidence IDs in the answer or at least in the trace. If it cannot identify which source supports a claim, the evidence is not ready for a customer-facing response.

Set Loop Limits and Stop Conditions

An agentic RAG loop without stopping rules becomes an expensive wander. Set limits before launch:

max_iterations: 2 or 3 for most interactive products
max_tool_calls_per_iteration: 2
max_total_tool_calls: 4
max_evidence_items: 8
max_latency_ms: product-specific p95 target
max_additional_cost_per_iteration: budget-specific
min_new_evidence_to_continue: 1

Continue only when all three conditions are true:

1. The previous answer would be incomplete.
2. The system can name the missing information.
3. There is an approved tool or retrieval path likely to provide it.

Otherwise answer, ask a human, or refuse. A safe refusal is better than another confident synthesis from weak context.

def should_continue(state: RetrievalState, judgment: dict) -> NextAction:
    if state.iterations >= state.max_iterations:
        return NextAction.ANSWER

    missing = judgment.get("missing_information", [])
    if not missing:
        return NextAction.ANSWER

    if len(state.evidence) >= 8:
        return NextAction.ANSWER

    if judgment.get("confidence", 0.0) >= 0.80:
        return NextAction.ANSWER

    if any(needs_human(item) for item in missing):
        return NextAction.ASK_HUMAN

    return NextAction.RETRIEVE

Every continue decision should append a reason to the trace. "Another retrieval" is not a reason; "missing refund eligibility for invoice 88123" is.

Build the Full Agentic RAG Loop

The following controller shows the shape without hiding the decisions behind a framework. Replace the helper functions with your own retrievers, model adapter, and authorization layer.

def run_agentic_rag(state: RetrievalState, principal: dict) -> dict:
    state.next_action = classify_query(state.original_query)
    state.working_query = state.original_query

    while state.iterations < state.max_iterations:
        state.iterations += 1

        if state.next_action == NextAction.ANSWER:
            break

        if state.next_action == NextAction.REWRITE:
            rewrite = call_query_rewriter(state)
            state = rewrite_query(state, rewrite)
            state.next_action = NextAction.RETRIEVE

        if state.next_action == NextAction.DECOMPOSE:
            plan = call_query_planner(state)
            state.subqueries = plan.get("subqueries", [])[:2]
            state.next_action = NextAction.RETRIEVE

        if state.next_action == NextAction.USE_TOOL:
            tool_name = plan_tool(state, principal)
            tool = TOOLS[tool_name]

            if principal["role"] not in tool.allowed_roles:
                return {
                    "status": "blocked",
                    "reason": "tool_not_allowed",
                    "trace_id": state.request_id,
                }

            result = tool.handler(
                **tool_arguments_for(tool_name, state),
                owner_id=state.owner_id,
            )
            state.tool_trace.append(
                {
                    "tool": tool_name,
                    "returned_items": len(result),
                    "iteration": state.iterations,
                }
            )
            state.evidence.extend(result)

        if state.next_action == NextAction.RETRIEVE:
            queries = state.subqueries or [state.working_query]
            for query in queries:
                candidates = vector_search(
                    collection="knowledge_base",
                    query=query,
                    owner_id=state.owner_id,
                    top_k=5,
                )
                state.evidence.extend(candidates)

        selected = deduplicate_evidence(state)
        judgment = judge_sufficiency(state, selected)

        state.missing_information = judgment.get("missing_information", [])
        state.next_action = should_continue(state, judgment)

        # REFUSE and ASK_HUMAN are terminal: the loop must exit immediately.
        if state.next_action == NextAction.REFUSE:
            return {
                'status': "refused",
                'reason': "insufficient_or_unsafe_context",
                'trace_id': state.request_id,
            }

        if state.next_action == NextAction.ASK_HUMAN:
            return {
                'status': "escalated",
                'reason': "requires_human_review",
                'trace_id': state.request_id,
            }

        if state.next_action != NextAction.RETRIEVE:
            break

    selected = deduplicate_evidence(state)
    answer = synthesize_answer(state.original_query, selected)

    return {
        "status": "answered",
        "answer": answer,
        "evidence_ids": [item.evidence_id for item in selected],
        "iterations": state.iterations,
        "trace_id": state.request_id,
    }

The agentic RAG LangGraph pattern can express this same loop as a graph. The advantage is not magic; it is a repeatable way to define nodes, edges, and state transitions. If your team already has a workflow engine, you can implement the same pattern there. Do not adopt a framework before you can write the state and decisions on paper.

Route Models by Task, Not by Hype

Agentic RAG uses models for several jobs: routing, rewriting, judging evidence, tool planning, and final synthesis. These tasks have different latency and quality requirements, so one model route is rarely optimal.

For example, a team might evaluate GPT-5.6 Terra for multi-hop synthesis, DeepSeek V4 Pro for analytical or code-adjacent retrieval, and GLM 5.2 for multilingual or summary workloads where it meets quality requirements. These are candidate routes, not universal recommendations. Verify exact model IDs and feature support in the live OurToken model catalog.

The LLM model routing guide explains how to route simple work to faster routes and complex work to stronger routes. In agentic RAG, routing has one additional rule: the loop must still stop safely when a routing call fails. A missing classifier should fall back to conservative retrieval, not to an unauthorized tool.

Because RAG tool calling is central to the loop, review the OpenAI-compatible tool calling guide for request mechanics. Keep tool policy in your application; compatibility does not mean the model should decide what the caller is allowed to access.

Evaluate Decision Quality, Not Only Answer Quality

A traditional RAG evaluation can say that an answer was unfaithful. An agentic RAG evaluation should also ask whether the system made the right retrieval decisions.

Build a fixed evaluation set with:

• Questions that should not retrieve.
• Simple lookups that should retrieve once.
• Multi-hop questions requiring decomposition.
• Questions that need a database tool instead of prose.
• Cases where evidence conflicts.
• Cases where the correct behavior is a refusal or human handoff.

The RAG evaluation metrics guide covers faithfulness, relevance, context precision, and context recall. For agentic RAG, add the planner metrics above. A system can improve answer quality while becoming much more expensive because it retrieves three times instead of once; without loop metrics, that regression looks like progress.

Evaluate Retrieval Components Separately

Do not change the embedding model, chunking policy, reranker, prompt, and model route in one experiment. Freeze one layer, measure the next, and save retrieved source IDs. The embedding model comparison guide is useful when you are comparing embedding routes, but the same discipline applies to the retrieval controller.

A useful comparison table looks like this:

case_id
query_class
expected_action
actual_action
correct_tool
used_tools
retrieved_source_ids
expected_source_ids
iterations
answer_faithfulness
answer_relevance
latency_ms
input_tokens
output_tokens
cost_usd
stop_reason

Then inspect failures by class. An agentic loop may be excellent at single-policy questions and bad at time-sensitive account status. Aggregate scores alone will not show that.

Combine Agentic RAG with Memory and Guardrails

Some requests need prior context: "Did the same issue happen again?" or "Use the workaround from yesterday's incident." That requires controlled memory, not a transcript dump.

The AI agent memory architecture guide separates working memory, vector memory, and summary memory. Apply the same idea to retrieval: keep the current task in working state, retrieve episodic evidence when relevant, and load durable summaries only when they pass tenant, consent, freshness, and permission checks.

Production Checklist

Before exposing an agentic RAG workflow, verify:

[ ] The system can decide not to retrieve.
[ ] Query rewrites retain the original query and tenant scope.
[ ] Subqueries are bounded and independently useful.
[ ] Every tool has an owner scope, role policy, schema, and timeout.
[ ] Tool calls are validated and authorized before execution.
[ ] Evidence has source IDs, timestamps, scores, and access scope.
[ ] Similarity is combined with authority, freshness, and entity match.
[ ] Evidence is deduplicated before generation.
[ ] Sufficiency checks return typed missing-information fields.
[ ] Loop iterations, tool calls, evidence size, latency, and cost are capped.
[ ] Every retrieval, tool, continue, and stop decision is traced.
[ ] Model routes have fallbacks and cannot bypass deterministic permissions.
[ ] Evaluation includes no-retrieve, single-hop, multi-hop, conflict, and refusal cases.
[ ] Cost per successful answer is measured, not just average answer quality.

Conclusion

Agentic RAG becomes useful when retrieval acquires judgment. The system should decide whether retrieval is necessary, how to rewrite a request, which protected tool can supply missing facts, whether evidence is sufficient, and when to stop.

Start with one workflow where a fixed pipeline already fails: support investigations, account-specific questions, incident summaries, or multi-document comparisons. Build the loop around a small state object, protect every tool, judge evidence with deterministic and model-assisted signals, and evaluate decision quality alongside answer quality. With those boundaries in place, an agentic retrieval loop becomes controllable instead of merely autonomous.

Related OurToken Guides

• AI agent guardrails
• AI agent memory architecture
• Embedding model comparison
• LLM model routing
• OpenAI-compatible tool calling
• RAG evaluation metrics
• OurToken model catalog

FAQ

What is agentic RAG?

Agentic RAG is a retrieval architecture where the system decides when to retrieve, how to rewrite or decompose a query, which retrieval tools to use, whether evidence is sufficient, and when to stop iterating. It is more than a fixed embed-fetch-generate pipeline.

How is agentic RAG different from traditional RAG?

Traditional RAG follows the same retrieval path for every request. Agentic RAG uses conditional paths based on query type, available tools, evidence quality, access scope, and stop conditions. It also records why those paths were chosen.

Do I need LangGraph for agentic RAG?

No. LangGraph can be a good way to express nodes and state transitions, but the core pattern is a state machine with retrieval, evidence judgment, tool policy, and stop rules. A simple workflow engine can implement the same idea.

When should a RAG system call tools instead of vector search?

Use tools when the answer depends on current state, exact entity lookup, permissions, or structured fields. Vector search is strong for prose and policies; it is usually the wrong source for payment status, deployment state, quotas, or account settings.

How many retrieval iterations should an agentic RAG loop run?

For interactive products, start with two or three iterations, one or two tool calls per iteration, and a hard evidence cap. Continue only when the system can name missing information and has an approved path likely to provide it.

How do I evaluate agentic RAG?

Evaluate both answer quality and decision quality. Track unnecessary retrieval, intent preservation after rewrite, tool correctness, evidence coverage, conflicts, iterations, stop reason, latency, tokens, and cost per successful answer.

Can agentic RAG reduce cost?

It can, but not automatically. Routing easy requests away from retrieval and reserving loops for complex cases may reduce waste. Poorly bounded loops can cost much more than traditional RAG, so measure cost per successful answer and cap iterations.

What safety controls does agentic RAG need?

Every proposed tool call needs argument validation, tenant checks, role checks, rate limits, timeouts, and audit logging. Retrieval and memory must respect access scope, and the model should never be allowed to bypass deterministic permissions.