智能体
dev.to ·
@Riven Desk
AI agent 声称测试通过后，应该核查什么
文章主张 agent 给出的绿色 CI 不是合并信号，而只是一个“我运行了某命令且退出码为零”的声明。作者提出四步核查顺序：1）先看实际改动的文件列表，判断 diff 是否超出工单范围、是否夹带 lockfile 或无关配置；2）对抗性阅读断言，检查回归重现时测试是否仍会通过，警惕只断言“有返回”或状态码 200、快照吸收行为变化、仅覆盖 happy path；3）主动验证失败路径，如无效输入、缺失鉴权、权限拒绝、空集合、超时、部分写入、feature flag 关闭与幂等性；4）在本地或同一 CI 任务中用 PR 分支复现 agent 声称运行的命令，确认命令、工作目录与环境假设一致，并排查 flaky、skip 和“带警告通过”。作者强调顺序为 diff→断言→失败路径→本地复现，绿色是必要条件而非充分条件。文末推广其 AI Agent Code Review Kit。
AI推荐理由 文章给出了一套针对 AI agent 声称“测试通过”后的具体核查流程：先看 diff 文件列表而非 agent 叙述、对抗性阅读断言（是否能在回归重现时仍通过）、覆盖失败路径（无效输入、权限拒绝、超时、幂等）、本地复现同一命令。这些是可操作的工程实践，对使用 coding agent 的开发者有直接价值。局限在于内容为个人经验总结，无实证数据或工具对比，且结尾带有付费 Code Review Kit 的推广链接，存在营销成分。
72
正文
Green CI from an agent is not a merge signal. It is a claim: "I ran something and it exited zero." Your job is to check whether that claim covers the bug, the intent, and the failure modes you care about.
I treat "tests pass" as the start of a short verification sequence — not the end of review.
1. Inspect the diff, not the summary
Open the file list first. Ignore the agent's narrative until you can answer:
Which files actually changed?
Do they match the human intent in one sentence?
Any lockfiles, renames, config, or test fixtures you did not ask for?
Agents often pad the suite while touching unrelated helpers. If the diff is wider than the ticket, pause before you trust the green check. A passing suite on the wrong surface is still the wrong merge.
2. Read the assertions (adversarially)
Open the new or edited tests and ask: would this fail if the original bug came back?
Watch for:
Asserts that only check "something returned" or status 200
Snapshots that absorb any behavior change
Happy-path-only coverage with no edge or negative case
Tests that fail solely if the function is deleted
If the assertion would still pass with the regression restored, the suite is theater. Request a tighter assert before you approve. Prefer one sharp negative case over five soft positives.
3. Exercise the failure paths
Green tests often skip the paths that hurt in production:
Invalid input, missing auth, permission denied
Empty collections, timeouts, partial writes
Feature-flag off, second call, idempotency
Pick the failure mode closest to the ticket and ask whether any test forces it. If not, either add that case or manually exercise it before merge. Agents optimize for "looks covered." You optimize for "breaks when broken."
4. Reproduce the command locally
Do not trust the agent's pasted output alone. Run the same command on your machine (or the same CI job) with the PR branch checked out:
# example — use whatever your repo actually runs
npm test -- path/to/relevant.spec.ts
Enter fullscreen mode
Exit fullscreen mode
Check:
Same command the agent claimed to run?
Same working directory / env assumptions?
Flakes, skips, or "passed with warnings" you would not accept?
If you cannot reproduce green locally, you do not have a pass — you have a story. Fix the story before merge.
Sequence, not vibes
Order matters: diff → assertions → failure paths → local reproduce. Skip ahead and you rubber-stamp confidence. Stop early when the file list or asserts are weak do not sink twenty minutes into a suite that never could catch the bug.
This is the same bar I use on agent PRs elsewhere: keep the speed, keep your judgment. Green is necessary. It is not sufficient.
If you want the packaged checklist, Cursor-oriented rules, and review prompts I use on agent PRs, the AI Agent Code Review Kit is here: https://chopragunji.gumroad.com/l/nxoboi
— Riven Desk
What do you check first after an agent claims tests pass — file list, asserts, or a local re-run? Drop your sequence in the comments.