RAG与Agentic RAG：Agent究竟何时真正有用

Artificial Intelligence Agents Agentic Rag Rag Machine Learning RAG vs Agentic RAG: When Does an Agent Actually Help? Understanding when traditional RAG is enough — and when your system needs to reason, retrieve, and decide. Garvit Agarwal 9 min read · 22 hours ago -- Listen Share You built a RAG system. A user asks a question. Your application searches the vector database, retrieves the most relevant chunks, sends them to the LLM, and generates an answer. Everything works. Sounds good…. Then someone asks: “Compare our product performance across the last two quarters, identify where we are losing to competitors, check whether engagement dropped in those regions, and tell me what we should investigate next.” Suddenly, retrieving the top-k most similar chunks doesn’t feel so intelligent anymore. The question isn’t asking for one piece of information. It is asking the system to find information, decide what else it needs, potentially search multiple sources, compare results, and reason through several steps. Press enter or click to view image in full size When a simple RAG question becomes a multi-step investigation. So, should we simply improve our RAG pipeline? Or does this require an AI agent? That brings us to an increasingly important distinction: RAG retrieves context. Agentic RAG decides how to retrieve it. And that difference matters much more than the terminology suggests. First, What Problem Was RAG Trying to Solve? Large Language Models know a lot, but they don’t automatically know your organization’s latest documents, internal databases, proprietary information, or information created after their training. This is where Retrieval-Augmented Generation (RAG) comes in. The original RAG research introduced the idea of combining a model’s parametric knowledge with an external, non-parametric knowledge source. In practical applications today, the architecture often looks something like this: User Question -> Create Query / Embedding -> Search Knowledge Base -> Retrieve Relevant Context -> Send Question + Context to LLM -> Generate Answer Suppose an employee asks: “What is our company’s parental leave policy?” A RAG system can search the HR knowledge base, retrieve the relevant policy section, and provide it to the LLM. The model now answers using information it did not necessarily contain in its training data. This ability to ground an LLM in private or frequently changing information is one of the primary reasons RAG became such an important architecture for enterprise Generative AI. But notice something about this workflow. The retrieval strategy has already been decided by the developer. The application essentially says: When a question arrives → search this knowledge base → retrieve N documents → send them to the model. The LLM usually doesn’t decide: Do I actually need retrieval? Which database should I search? Was the first search sufficient? Should I reformulate the query? Do I need information from another system? Should I perform another retrieval before answering? The pipeline determines most of that beforehand. And for many applications, that’s perfectly fine. Press enter or click to view image in full size Traditional RAG follows a straightforward Retrieve → Generate workflow. So Where Does Traditional RAG Start Struggling? Consider this question: “Which products had declining sales last quarter, whether customer complaints increased for those products, and whether any of them currently have open recalls?” Now imagine your information lives in three places: Database A: Sales data Database B: Customer-support records External API: Product recall information A simple similarity search can’t completely solve this. The system first needs to determine which products experienced declining sales. Only after discovering those products does it know what customer-support information to retrieve. And after that, it needs to check the recall source for those specific products. The retrieval path might therefore look like: Question → Find products with declining sales → Identify affected products → Search support records for those products → Check recall database/API → Compare results → Generate response The important difference is that the result of one step determines what happens next. A fixed retrieval pipeline becomes increasingly difficult to design as these possibilities multiply. This is where Agentic RAG becomes interesting. What Is Agentic RAG? In Agentic RAG, retrieval becomes something the model can decide to use as a tool, rather than simply being a mandatory fixed step before generation. A simplified architecture looks like: User Question ↓ Agent / LLM ↓ Plan ↓ Choose Tool ↙ ↓ ↘ Vector Search | SQL | API ↓ Evaluate Result ↓ Enough information? No → Search/Act Again Yes → Generate Answer Microsoft describes standard RAG as an orchestrator following a fixed sequence — accept the query, search, assemble context, and invoke the model. In an agentic architecture, the agent can instead determine whether to search, which source to query, and whether additional retrieval is necessary. That makes retrieval dynamic. Press enter or click to view image in full size Agentic RAG adds decision-making to the retrieval process. The Key Difference: Retrieval vs Decision-Making Here is perhaps the simplest way to think about it. Traditional RAG: Retrieve → Generate Agentic RAG: Think → Retrieve → Evaluate → Think → Retrieve/Act → Generate Traditional RAG asks: “What information is similar to this question?” Agentic RAG can ask: “What information do I need to solve this question, where should I get it, and do I have enough information yet?” That additional decision layer is the real reason to introduce an agent. Example: Where the Difference Becomes Obvious Imagine we are building an AI assistant for a pharmaceutical company. A Brand Director asks: “Why did Product A lose market share in the West region over the last four weeks?” The answer might depend on several datasets: prescription data competitor performance HCP engagement sales calls payer/access data A traditional RAG implementation might search documents related to: “Product A market share West region last four weeks.” It retrieves the most semantically relevant information and asks the LLM to answer. But the actual investigation could require something more sophisticated. An agent could reason: Step 1: I need to verify whether market share actually declined. → Query prescription/sales data. Step 2: The decline appears concentrated in three territories. → Query those territories specifically. Step 3: Did competitor prescriptions increase? → Query competitor data. Step 4: Competitor prescriptions increased significantly. → Check whether HCP engagement changed. Step 5: Engagement declined among several high-value HCPs. → Check payer/access information for another possible explanation. Only then does the system synthesize the evidence. Notice what changed. We didn’t necessarily create a smarter retriever. We created a system capable of deciding what to investigate next. Query Decomposition Is a Big Part of Agentic RAG Real questions are rarely perfectly formatted search queries. Get Garvit Agarwal’s stories in your inbox Join Medium for free to get updates from this writer. Subscribe Subscribe Remember me for faster sign in Consider: “Compare the performance of Product A and Product B in the North and West regions and explain what changed.” One giant vector-search query may produce mediocre retrieval. An agent could decompose it into smaller questions: Query 1: Product A performance — North Query 2: Product A performance — West Query 3: Product B performance — North Query 4: Product B performance — West Then compare the retrieved evidence. Modern agentic retrieval systems increasingly incorporate this idea. Microsoft’s current RAG documentation, for example, describes agentic retrieval as using a model to break complex inputs into multiple focused subqueries and potentially execute them in parallel. This matters because sometimes the problem isn’t that your vector database couldn’t find the answer. The original question was simply a bad retrieval query. Press enter or click to view image in full size The agent uses intermediate results to decide the next retrieval step. Another Important Capability: Iterative Retrieval Suppose the first retrieval returns weak evidence. Traditional RAG might simply send those chunks to the LLM anyway. An agentic system can potentially recognize: “I still don’t have enough evidence.” Then try another query. For example: Initial query: “Why did revenue decrease?” Results aren’t sufficient. Refined query: “Revenue decline Q2 enterprise customers APAC” Still incomplete. Search another data source. Retrieve customer churn information. Now generate the answer. This creates a retrieval loop: Search → Inspect → Refine → Search Again rather than: Search Once → Hope for the Best. But Here’s the Important Part: Agentic RAG Is Not Automatically Better This is where I think many AI architectures become unnecessarily complicated. A team starts with: LLM Then adds: RAG Then: Agent Then: Multi-Agent System And eventually you have six agents discussing a question that could have been answered with one database query. More intelligence in the architecture does not automatically mean a better product. Microsoft’s architecture guidance explicitly notes that standard RAG remains appropriate when questions can be handled through a single search against a single index. Agent reasoning steps introduce additional latency, token consumption and system complexity. IBM similarly notes that agentic RAG can introduce additional token costs and latency and that agents themselves are not perfectly reliable. That trade-off deserves more attention. When Traditional RAG Is Probably Enough Imagine your chatbot primarily handles questions such as: “What is our reimbursement policy?” “What does this product documentation say about authentication?” “Summarize our onboarding guidelines.” “What are the eligibility requirements?” These questions generally follow: Question → Find relevant document → Answer There isn’t much benefit in giving an LLM the autonomy to construct a multistep plan. You may get more value from improving: Chunking, Embedding quality, Metadata filtering, Hybrid search, Reranking, Prompt design before introducing an agent. Microsoft’s Azure AI Search guidance, for example, recommends techniques including hybrid keyword/vector retrieval and semantic ranking to improve RAG relevance. Sometimes what looks like a need for “Agentic RAG” is actually just a retrieval-quality problem. When Does an Agent Actually Help? Agentic RAG starts becoming useful when the retrieval path cannot easily be known beforehand. There are several strong signals. 1. The question requires multiple steps The answer to one search determines the next search. Find X → use X to find Y → compare Y with Z. 2. You have multiple data sources Your system may need to choose between: Vector database, SQL database, REST API, Knowledge graph, Web search, Internal documents Instead of querying everything every time, an agent can select the appropriate source. 3. Complex questions need decomposition A single user request may actually contain four or five retrieval problems. The agent can break the request into smaller searches and combine their results. 4. Retrieval may need refinement The first search doesn’t always return sufficient evidence. The agent can modify its search strategy and try again. 5. Retrieval needs to lead to an action Imagine: “Check whether this customer’s order qualifies for a refund and initiate it if eligible.” Now we’re no longer doing only: Retrieve → Answer We’re doing: Retrieve policy → Retrieve order → Evaluate eligibility → Call refund API → Confirm action That is much closer to an agentic workflow. Press enter or click to view image in full size An agent earns its place when retrieval becomes a decision-making problem. RAG vs Agentic RAG Press enter or click to view image in full size Traditional RAG follows a predefined path; Agentic RAG can dynamically decide what happens next. A Practical Rule I Use Before adding an agent, ask: Can I determine the retrieval steps before the user asks the question? If the answer is yes, you probably don’t need an agent. Build a good RAG pipeline. If the answer is: “It depends on what the system discovers along the way.” then Agentic RAG starts making much more sense. That single distinction prevents a surprising amount of unnecessary architecture. The Bigger Shift RAG originally changed the question from: “What does the model know?” to: “What information can we retrieve for the model?” Agentic RAG changes the question again: “Can the model determine what information it needs next?” That’s a much bigger architectural shift than simply adding another framework around an LLM. Traditional RAG gives the model knowledge. Agentic RAG can give the model choices about how to acquire that knowledge. And choices are useful only when there is actually a decision worth making. Final Thought Before turning your RAG application into an agent, ask yourself: Does my system need better retrieval — or does it genuinely need decision-making? If users ask straightforward questions against a well-defined knowledge base, a well-designed RAG pipeline may be faster, cheaper, easier to evaluate, and easier to maintain. But when questions require the system to decompose problems, choose among multiple sources, retrieve iteratively, evaluate intermediate results, and decide what to do next, an agent starts earning its place in the architecture. Don’t add an agent because RAG feels old. Add an agent when the next retrieval step cannot be predetermined. And perhaps that’s the more useful question than “RAG or Agentic RAG?” At what point does your retrieval problem become a decision-making problem? Comment down your thoughts and clap if you like it.