智能体
dev.to ·
@Shaheer Ul Hassan
在 LangGraph 中构建 rag-critic：事实接地与对抗式幻觉审计
作者指出生产 RAG 的主要故障是静默幻觉，而标准做法用 LLM 对整段答案做评判，存在反馈粒度粗、延迟高等问题。为此他发布 PyPI 包 rag-critic，面向 LangGraph 循环多智能体图提供轻量的对抗式幻觉审计与引用接地引擎，可定位具体句子是否缺乏检索片段支持。
AI推荐理由 作者发布 PyPI 包 rag-critic，针对 LangGraph 多智能体图做句子级幻觉审计与引用接地，指出段落级 LLM 评判反馈粗糙、延迟高的问题；有具体工具与架构，但效果仅有作者自述，缺少独立评测。
66
正文
When building production Retrieval-Augmented Generation (RAG) pipelines, the primary failure mode isn't retrieval latency or vector index size—it's silent hallucination.
LLMs frequently generate fluent, highly plausible-sounding claims that are completely unsupported by the retrieved reference chunks. In multi-agent systems and mission-critical workflows, accepting an ungrounded draft can corrupt downstream decision-making.
To solve this in cyclic multi-agent graphs, I built and published rag-critic on PyPI—a lightweight, adversarial hallucination auditor and citation grounding engine architected specifically for LangGraph.
🔍 The Architectural Bottleneck
Standard RAG architectures evaluate answers at a holistic paragraph level using an LLM-as-a-judge. This approach has three major flaws:
Coarse-Grained Feedback: Telling an agent "this answer feels inaccurate" doesn't specify which sentence was hallucinated.
High Latency: Re-prompting large frontier models for full evaluations adds 3–5 seconds per step.
Lack of Determinism: Without an entailment matrix, confidence scores drift across model runs.
📦 Project Links & Open-Source Repositories
PyPI Package: pypi.org/project/rag-critic
GitHub Repository: github.com/samjanjua6/rag-critic
Author Engineering Portfolio: https://shaheerulhassan.com
💻 Quickstart & Python Usage
Install directly from PyPI:
bash
pip install rag-critic
from rag_critic import RAGCritic
critic = RAGCritic()
# Evaluate a draft answer against retrieved source chunks
result = critic.evaluate(
draft_response="Python was released in 1991 by Guido van Rossum. It was originally named Monty.",
retrieved_chunks=[
"Python is an interpreted programming language released in 1991 by Guido van Rossum."
]
)
print(f"Status: {result.status}") # REVISE / APPROVED
print(f"Grounding Score: {result.grounding_score:.2f}") # e.g. 0.50
print(f"Hallucination Index: {result.hallucination_index:.2f}")
# Inspect claim-level verification
for claim in result.claims:
print(f"[{claim.status.value}] {claim.text}")
# [SUPPORTED] Python was released in 1991 by Guido van Rossum.
# [UNGROUNDED] It was originally named Monty.
from langgraph.graph import StateGraph, END
from rag_critic import create_critic_node, critic_router
builder = StateGraph(MyRAGState)
# Add standard generator and retriever nodes
builder.add_node("retrieve", retrieve_docs)
builder.add_node("generate", generate_draft)
# Add rag-critic as an adversarial validation checkpoint
builder.add_node("critic", create_critic_node())
builder.add_edge("retrieve", "generate")
builder.add_edge("generate", "critic")
# Dynamic cyclic routing based on hallucination thresholds
builder.add_conditional_edges(
"critic",
critic_router,
{
"approved": END, # Output is grounded; return to user
"rewrite": "generate", # Minor gaps; instruct re-writer
"re_retrieve": "retrieve", # Heavy hallucination; expand search
}
)
Enter fullscreen mode
Exit fullscreen mode