行业动态
hackernoon.com ·
@trpevski
如何使用RAGAS评估代理型RAG
本文展示了使用RAGAS、Pinecone和Cohere对工具调用型RAG代理进行评分的过程，发现低检索分数实为语料库覆盖不足所致，而非检索质量问题。
AI推荐理由 具体介绍了使用RAGAS、Pinecone和Cohere评估工具调用型RAG代理的方法，并指出低检索分数源于语料库覆盖问题而非检索质量，对RAG实践有参考价值。但来源为技术博客，非官方文档，且未提供详细实验数据。
70
正文
如何用RAGAS评估Agentic RAG
作者：
trpevski
2026年9月3日
TLDR
您的浏览器不支持音频元素。
作者：trpevski@trpevski
作者：trpevski|@trpevski
我是Darko Trpevski，一名软件工程师和架构师，拥有超过16年的生产软件构建经验。
故事的可信度
我想看看RAG评估实际上是如何工作的，所以我构建了一个小型工具调用代理，并用RAGAS对其进行评分。指标是最容易的部分。
RAGAS分别对检索和生成进行评分，这种区分在代理管道中最为重要：当代理决定完全不搜索时，contexts字段会返回空，无论答案是否实际良好，两个检索指标都会使该行失败。我在Pinecone上使用Cohere嵌入构建了一个小型工具调用代理，并用RAGAS对其进行端到端评分，以准确找出问题所在。
RAGAS分别对检索和生成进行评分，这种区分是重点。它告诉您应该修复哪一半，而不是只给您一个数字。
设置很简单：Cohere用于嵌入，Pinecone用于索引，一个OpenAI代理将检索作为工具调用。没有什么特别的。
有趣的问题不在于指标。而在于当代理决定完全不搜索时，contexts字段中没有任何内容，检索分数会变得奇怪。
设置
故意保持小型，因为重点是测量而不是管道。一个公开的预分块ArXiv论文数据集，过滤到其中两篇：Llama 2和Mixtral of Experts。Cohere嵌入块，Pinecone存储它们，一个OpenAI代理获得一个工具：搜索该索引。
这两篇论文不是随机选择的。我使用的评估问题是关于这些论文的，只有当答案实际在索引中时，检索才能看起来良好。一旦你说出来就显而易见，但在组装演示时很容易出错。
# 代理决定何时调用此函数。事实证明这很重要
@tool
def search_arxiv(query: str) -> str:
"""搜索索引的ArXiv论文以查找与查询相关的段落。"""
vec = embed.embed_query(query)
hits = index.query(vector=vec, top_k=5, include_metadata=True)
return "\n\n".join(h["metadata"]["text"] for h in hits["matches"])
索引使用余弦相似度，因为它比较方向而不是长度，嵌入幅度通常只反映文本的长度。
完整代码，包括评估脚本，在GitHub上：agentic-rag-evaluation-ragas。
RAGAS实际关注什么
这是我发现真正有用的部分，也是为什么我会再次使用RAGAS而不是编写自己的检查。
它分别对两件事进行评分。检索：搜索是否找到了正确的材料。生成：考虑到找到的内容，答案是否良好。这些失败的原因不同，修复方法也不同，因此单一的合并分数几乎无用。
在检索方面，上下文精确度询问返回的内容中有多少是实际相关的，而上下文召回率询问需要的内容中有多少实际返回了。噪声与不完整。
在生成方面，忠实度将答案分解为单个声明，并针对检索到的文本检查每个声明，这是幻觉检查。答案相关性询问它是否回答了问题，无论是否正确。答案正确性则直接与真实答案进行比较。
在运行之前值得知道的一件事：大多数这些指标使用模型作为评判者，因此评估涉及大量API调用。这不是可以在每次提交时随意进行的免费操作。
每个指标得分低可能意味着什么，以及尝试什么：
上下文召回率——可能索引中缺少内容，或top_k太小。尝试更小的块、更多的块、更广泛的语料库。
上下文精确度——可能检索有噪声。尝试重新排序，或更严格的阈值。
忠实度——可能模型超出了其上下文。尝试收紧提示，降低温度。
答案相关性——可能是提示问题，而不是检索问题。尝试重写指令。
仅答案正确性——可能是实质问题，或真实答案本身。自己阅读该行。
然后代理决定不搜索
这是变得有趣的地方，也是我停止在脑海中跟随教程的地方。
每个人都使用的评估格式是问题、检索到的上下文、答案、真实答案。对于固定管道（总是先检索然后生成）来说没问题。但我的代理可以选择。有时它只是回答，因为模型已经对Llama 2了解足够多，可以在不搜索的情况下说出合理的内容。
当这种情况发生时，contexts字段为空，因此两个检索指标都将其评为失败。而这个问题可能已经完美回答了。将这些行平均到其他所有内容中，您的检索数字会说有些东西坏了，而实际上什么也没运行。
当代理使用不同查询搜索两次时，同样的问题以不同形式出现。您是给找到的所有内容评分，还是最后一次搜索，还是每次单独评分？三个答案，三个不同的数字，没有告诉您哪个是正确的。
还有一个我没有预料到的较小问题：代理在搜索前重写了问题。因此上下文精确度是根据用户的措辞来评判检索，而检索器实际看到的是代理的版本。如果精确度低，是搜索不好还是重写不好？指标无法告诉您。
我最终做的事情很简单，也不太令人满意。记录每个问题上工具是否被调用，以及调用次数，然后分别查看这些组，而不是将所有内容平均成一行数字。
数字说明了什么
两个视图，您都需要：
metric_cols = ["context_precision", "context_recall",
"faithfulness", "answer_relevancy", "answer_correctness"]
result_df[["user_input"] + metric_cols] # 每个问题，每行一个
result_df[metric_cols].mean() # 您向人们展示的摘要
每个
问题的RAGAS分数，带有一列显示代理是否调用了检索工具
整个运行的平均分数：
context_precision — 0.22
context_recall — 0.52
faithfulness — 0.75
answer_relevancy — 0.48
answer_correctness — 0.48
不是好数字，它们的形状比水平更有趣。
忠实度最高，为0.75，两个检索指标最低。这已经告诉您生成方面不是问题：当模型被给予可用的内容时，它保持在内容范围内。大多数行的context_precision正好为0。
找到问题实际所在
平均值说检索。行说更具体的内容。
几个问题返回干净：recall 1.0，faithfulness 1.0，relevancy约0.87。所有这些都是关于Mixtral of Experts对模型大小和令牌生成的影响。当材料在索引中时，整个管道工作正常。
然后有一组answer_relevancy正好为0.00。那里的精确零不是稍微偏离的答案，而是非答案，模型说它找不到信息。这些行中的每一行也有context_precision为0。所以代理搜索了，没有获得有用的内容，并说了出来。
阅读这些问题询问的内容是理解的关键。专家LRU缓存。推测性专家加载。在A100上卸载。这些不在Llama 2中，也不在Mixtral of Experts中。它们来自另一篇关于MoE卸载的论文，而我的索引只包含我过滤的两篇。
因此，低检索分数主要是语料库覆盖问题，而不是检索质量问题。检索器被要求提供从未索引过的材料。
这种区别很重要，因为两者有相反的修复方法。检索质量差意味着重新排序、更好的嵌入、调整top_k。覆盖缺失意味着这些都没有帮助，您需要去索引论文。
有一行是值得指出的真正失败：context_recall 0.10，faithfulness 0.96，correctness 0.27。模型自信地回答，并且几乎不忠实于任何内容，这正是忠实度按文档工作，但也单独非常无帮助。忠实于糟糕的上下文仍然是错误的。
另一个是相反的：所有内容都接近零，包括faithfulness为0.125。一个量化问题，没有检索到任何内容，模型根据已知信息回答了。
这些都不是新思维。这是预测性ML一直需要的同样的两级习惯：聚合告诉您有变化，单个示例告诉您是什么。
花费我时间的小事
固定您的版本。LangChain在次要版本之间更改构造函数签名和导入路径，RAGAS比那更快。作为细粒度程度的体验，在RAGAS 0.3.x中，结果框架将问题列称为user_input而不是question，因此旧代码索引一个不再存在的列。
块大小是整个事情中最大的杠杆，在我的情况下，其他人已经拉动了它，因为数据集是预分块的。太大，每个嵌入混合多个主题，搜索变得模糊。太小，没有单个块包含完整答案。如果可能，值得自己拥有这个决定。
常见问题解答
RAGAS实际测量什么？分别测量检索和生成。上下文精确度和召回率检查搜索是否返回了相关且完整的材料。忠实度和答案相关性检查答案是否保持在材料范围内并解决了问题。答案正确性将答案与真实答案进行比较。
如何评估可能不检索的RAG代理？跟踪每个问题上是否调用了检索工具，并分别分组这些行。没有工具调用时contexts字段为空，因此检索指标使该行失败，无论答案是否良好，将它们混入平均值会使检索看起来比实际更差。
RAGAS分数产生成本吗？是的，大多数指标使用模型作为评判者，将答案分解为声明并检查每个声明。一次评估运行涉及大量调用，因此这不是可以不加思考地附加到每次提交的东西。
小评估集值得运行吗？对于找到要阅读的行，是的。对于您向某人引用的数字，不太值得。基于评判者的指标在运行之间变化，在小集上，一个坏行会明显移动平均值。
结论
连接起来是容易的部分。Cohere、Pinecone和一个带有一个工具的代理是一个下午的工作。
决定什么算公平分数花费了相当长的时间，我仍然想要一个好答案的是如何评估是否搜索的决定。一个跳过检索并幸运的代理，在这些指标中看起来与一个跳过检索并且正确跳过的一样。这感觉像是真正的问题，而五个指标都没有触及它。
关于作者
作者：trpevski|@trpevski
我是Darko Trpevski，一名软件工程师和架构师，拥有超过16年的生产软件构建经验。
阅读我的故事关于@trpevski
主题
ai-and-ml
#rag-and-ai#rag-evaluation#ai#artificial-intelligence#agentic-rag#tool-calling-agents#faithfulness#answer-correctness
查看原文（中文正文为 AI 翻译）
How to Evaluate Agentic RAG With RAGAS
by
trpevski
September 3rd, 2026
TLDR
Your browser does not support the audio element.
bytrpevski@trpevski
bytrpevski|@trpevski
I’m Darko Trpevski, a software engineer and architect with more than 16 years of experience building production software.
Story's Credibility
I wanted to see how RAG evaluation actually works, so I built a small tool-calling agent and scored it with RAGAS. The metrics were the easy part.
RAGAS scores retrieval and generation separately, and the split matters most on an agentic pipeline: when the agent decides not to search at all, the contexts field comes back empty and both retrieval metrics fail the row regardless of whether the answer was actually good. I built a small tool-calling agent over Pinecone with Cohere embeddings and scored it end to end with RAGAS to find exactly where that breaks.
RAGAS scores retrieval and generation separately, and that split is the whole point. It tells you which half to go fix instead of handing you one number.
The setup is straightforward: Cohere for embeddings, Pinecone for the index, an OpenAI agent that calls retrieval as a tool. Nothing exotic.
The interesting problem is not the metrics. It is that when the agent decides not to search at all, there is nothing to put in the contexts field and the retrieval scores go strange.
The setup
Deliberately small, because the point was the measurement and not the pipeline. A public dataset of pre-chunked ArXiv papers, filtered down to two of them, Llama 2 and Mixtral of Experts. Cohere embeds the chunks, Pinecone stores them, and an OpenAI agent gets one tool: search that index.
The two papers are not a random choice. The evaluation questions I used are about those papers, and retrieval can only look good if the answers are actually in the index. Obvious once you say it out loud, easy to get wrong when you are assembling a demo.
# the agent decides when to call this. that turns out to matter a lot
@tool
def search_arxiv(query: str) -> str:
"""Search the indexed ArXiv papers for passages relevant to a query."""
vec = embed.embed_query(query)
hits = index.query(vector=vec, top_k=5, include_metadata=True)
return "\n\n".join(h["metadata"]["text"] for h in hits["matches"])
Cosine similarity for the index, since it compares direction rather than length and embedding magnitude often just reflects how long the text was.
Full code, including the evaluation script, is on GitHub: agentic-rag-evaluation-ragas.
What RAGAS is actually looking at
This is the part I found genuinely useful, and it is why I would use RAGAS again rather than writing my own checks.
It scores two things separately. Retrieval: did the search find the right material. Generation: given what it found, is the answer any good. Those fail for different reasons and have different fixes, so a single combined score would be nearly useless.
On the retrieval side, context precision asks how much of what came back was actually relevant, and context recall asks how much of what was needed actually came back. Noisy versus incomplete.
On the generation side, faithfulness breaks the answer into individual claims and checks each against the retrieved text, which is the hallucination check. Answer relevancy asks whether it answered the question at all, regardless of whether it was right. And answer correctness compares against the ground truth directly.
One thing worth knowing before you run it: most of these use a model as judge, so evaluation is a lot of API calls. It is not a free operation you sprinkle on every commit.
What a low score on each metric probably means, and what to try:
Context recall — probably missing from the index, or top_k too small. Try smaller chunks, more chunks, a wider corpus.
Context precision — probably the retrieval is noisy. Try re-ranking, or a stricter threshold.
Faithfulness — probably the model going beyond its context. Try tightening the prompt, lowering temperature.
Answer relevancy — probably a prompt problem, not retrieval. Try rewriting the instructions.
Answer correctness only — probably substance, or the ground truth itself. Read the row yourself.
Then the agent decided not to search
Here is where it got interesting, and where I stopped following the tutorial in my head.
The evaluation format everybody uses is question, retrieved contexts, answer, ground truth. Fine for a fixed pipeline where you always retrieve then generate. But my agent gets to choose. Sometimes it just answers, because the model already knows enough about Llama 2 to say something reasonable without searching.
When that happens the contexts field is empty, so both retrieval metrics score it as a failure. On a question that might have been answered perfectly well. Average those rows into everything else and your retrieval numbers say something is broken when nothing actually ran.
Same thing in a different shape when the agent searches twice with different queries. Do you score everything it found, or the last search, or each one separately? Three answers, three different numbers, and nothing tells you which is correct.
And a smaller one I did not expect: the agent rewrites the question before searching. So context precision is judging retrieval against the user's wording, when the retriever actually saw the agent's version. If precision is low, is the search bad or is the rewrite bad? The metric cannot tell you.
What I ended up doing is simple and not very satisfying. Record whether the tool got called on each question, and how many times, then look at those groups separately instead of averaging everything into one row of numbers.
What the numbers said
Two views, and you want both:
metric_cols = ["context_precision", "context_recall",
"faithfulness", "answer_relevancy", "answer_correctness"]
result_df[["user_input"] + metric_cols] # every question, one row each
result_df[metric_cols].mean() # the summary you show people
Per-question RAGAS scores with a column showing whether the agent called the retrieval tool
Average scores across the run:
context_precision — 0.22
context_recall — 0.52
faithfulness — 0.75
answer_relevancy — 0.48
answer_correctness — 0.48
Not good numbers, and the shape of them is more interesting than the level.
Faithfulness is the highest at 0.75, and the two retrieval metrics are the lowest. That already tells you the generation side is not the problem: when the model was given something to work with, it stayed inside it. Most rows have context_precision at exactly 0.
Finding where a problem actually lives
The averages say retrieval. The rows say something more specific.
Several questions came back clean: recall 1.0, faithfulness 1.0, relevancy around 0.87. All of them are about what Mixtral of Experts does to model size and token generation. When the material was in the index, the whole pipeline worked.
Then there is a group where answer_relevancy is exactly 0.00. An exact zero there is not a slightly-off answer, it is a non-answer, the model saying it cannot find the information. Every one of those rows also has context_precision at 0. So the agent searched, got nothing useful, and said so.
Reading what those questions ask about is where it clicked. Expert LRU caching. Speculative expert loading. Offloading on an A100. Those are not in Llama 2 and they are not in Mixtral of Experts. They come from a different paper on MoE offloading, and my index only holds the two I filtered for.
So the low retrieval scores are mostly corpus coverage, not retrieval quality. The retriever was asked for material that was never indexed.
That distinction matters, because the two have opposite fixes. Bad retrieval quality means re-ranking, better embeddings, tuning top_k. Missing coverage means none of that helps and you go index the paper.
One row is a real failure worth naming: context_recall 0.10, faithfulness 0.96, correctness 0.27. The model answered confidently and stayed faithful to almost nothing, which is faithfulness working exactly as documented and also being deeply unhelpful on its own. Faithful to a bad context is still wrong.
Another is the opposite: everything near zero including faithfulness at 0.125. A quantization question, nothing retrieved, and the model answered anyway from what it already knew.
None of this is new thinking. It is the same two-level habit predictive ML always needed: the aggregate tells you something changed, the individual examples tell you what.
Small things that cost me time
Pin your versions. LangChain changes constructor signatures and import paths between minor releases and RAGAS moves faster than that. As a taste of how fine-grained it gets, in RAGAS 0.3.x the results frame calls the question column user_input rather than question, so older code indexes a column that is not there anymore.
Chunk size is the biggest lever in the whole thing and in my case somebody else had already pulled it, since the dataset came pre-chunked. Too large and each embedding blends several topics so search gets fuzzy. Too small and no single chunk holds a full answer. Worth owning that decision yourself if you can.
FAQ
What does RAGAS actually measure?Retrieval and generation separately. Context precision and recall look at whether the search returned relevant and complete material. Faithfulness and answer relevancy look at whether the answer stayed inside that material and addressed the question. Answer correctness compares the answer to a ground truth.
How do you evaluate a RAG agent that might not retrieve?Track whether the retrieval tool was called on each question and group those rows separately. With no tool call the contexts field is empty, so retrieval metrics fail the row regardless of whether the answer was good, and mixing them into the average makes retrieval look worse than it is.
Do RAGAS scores cost money to produce?Yes, most of the metrics use a model as judge, breaking answers into claims and checking each one. An evaluation run is a lot of calls, so it is not something to attach to every commit without thinking about the bill.
Is a small evaluation set worth running?For finding rows to read, yes. For a number you quote to somebody, not really. Judge-based metrics vary between runs and on a small set a single bad row moves the average visibly.
Conclusion
Wiring it up was the easy part. Cohere, Pinecone and an agent with one tool is an afternoon of work.
Deciding what counts as a fair score took considerably longer, and the thing I would still like a good answer for is how to grade the decision to search at all. An agent that skips retrieval and gets lucky looks identical, in these metrics, to one that skipped it and was right to. That feels like the actual question and none of the five metrics touch it.
About Author
bytrpevski|@trpevski
I’m Darko Trpevski, a software engineer and architect with more than 16 years of experience building production software.
Read my storiesAbout @trpevski
TOPICS
ai-and-ml
#rag-and-ai#rag-evaluation#ai#artificial-intelligence#agentic-rag#tool-calling-agents#faithfulness#answer-correctness