智能体
dev.to ·
@Aditya Mishra
Matrix：核验 AI agent 是否真的完成了它声称的操作
文章指出 agent 常对未发生的操作报告成功，且 trace 干净、无报错，现有可观测性工具因读取 agent 自述而误判为成功。Matrix 不重新解读 trace，而是查询权威系统（如真实 Gmail 邮箱），返回 confirmed / contradicted / inconclusive 并附证据，包括查询的账户、搜索窗口和实际结果。它区分两类失败：trace 中完全没有工具调用时，缺失本身即证据；调用发生且正常返回但无实际落地时，只有邮箱能说明问题。作者将 inconclusive 设为一等结论，在 trace 不完整、工具 span 名称未知或无法确定邮箱时拒绝判定。
AI推荐理由 针对 agent 自报成功但实际未执行的问题，提出不读 trace 而查询权威系统（如 Gmail 邮箱）并返回 confirmed/contradicted/inconclusive 及证据的校验思路，区分“无工具调用”与“调用成功但无落地”两类失败，对 agent 可观测性与可靠性有实际价值；但为个人项目介绍，缺少实现细节与评测数据。
70
正文
Agents report success for actions that never happened — no error, clean trace, and every observability tool reads it as a success, because they're all reading the agent's own account of itself.
This doesn't read the trace differently. It queries the authoritative system instead — the actual Gmail mailbox — and returns confirmed / contradicted / inconclusive with the evidence attached: the account queried, the search window, and what was found in it.
Two failure classes, different evidence. If there's no tool call in the trace at all, the absence is the evidence and nothing external is needed. If the call happened and returned cleanly but nothing landed, only the mailbox can tell you.
The parts I'd most want criticised:
Inconclusive is a first-class verdict. If the trace can't be shown complete, or the tool span has a name I don't recognise, or the mailbox can't be established — it refuses to judge rather than calling a working agent a liar. A false accusation costs more than a missed detection. I may have that balance wrong.
It cannot catch a correct call with a wrong argument. Asked to mail one person, confidently mails another — the send is real, Gmail confirms it, verdict is confirmed, correctly. The instruction is recorded next to the arguments so a human can see it. No verdict catches it.
Gmail only so far. LangChain via a callback handler, or the SDK by hand. No n8n, no CrewAI. Zero users — this is day one.
matrixverify.dev