你的 README 代码示例正在悄悄说谎——我用纯 AST（不用 LLM）做了个检测文档漂移的 CLI

Your README code examples are silently lying — I built a CLI to detect documentation drift using only AST, no LLM "Did you know? This README example doesn't work." Have you ever had a junior engineer tell you that? Or worse — have you added a line to your code, forgot the docs, and broke it yourself? This time I want to tell you about doc-drift, a CLI I built that detects this "documentation drift" — the silent gap between code examples in your Markdown and the actual codebase — without using an LLM at all. https://github.com/sunnydachs/doc-drift What it does You point it at a repository. It scans your Markdown files for fenced code blocks, extracts the functions and classes defined there, and checks whether they still match the real codebase. # scan the repo (read-only) doc-drift # another repo, machine-readable output doc-drift /path/to/repo --json Enter fullscreen mode Exit fullscreen mode Example output: doc-drift — scanned /home/dev/myproject markdown files: 42 | python blocks checked: 87 README.md:153 ⚠️ SIGNATURE DRIFT block defines (url, retries) but repository defines (url, timeout) summary: {"signature_drift": 3, "missing": 1, "unparseable": 5} | clean blocks: 41 Enter fullscreen mode Exit fullscreen mode It reports three main findings: SIGNATURE DRIFT — the documented function exists, but its arguments have diverged from the real code. MISSING — the documented function/class no longer exists anywhere in the repo (renamed? deleted?). UNPARSEABLE — the block isn't valid Python (pseudo-code, placeholders) — informational only. A "function exists, but arguments differ" bug is one that IDEs and static analysis can never catch. Docs aren't executed — so nothing in the toolchain ever checks them. That's the hole. Why deliberately not use an LLM This was the core design decision. An AI reading the natural-language docs could grasp more "context." But I deliberately built it on nothing but Python's standard ast (Abstract Syntax Tree). Three reasons: Deterministic. Same input → same output, every time. I don't want a tool in CI whose verdict flips between runs. Read-only and safe. It never imports or executes your code — it compares at the syntax-tree level. Your files stay untouched. Zero dependencies. Python 3.11+ is enough. Install is pip install git+... This is one answer to the question I keep asking myself when automating: how much should I hand to an AI? Deterministic work deserves deterministic tools. The rule: docs may omit, but they must never invent The detail I obsessed over: code examples in docs are almost always simplified versus the real code. Omitting an argument should be fine. So the rule is: Omit arguments — allowed. Invent arguments or functions that don't exist in the code — forbidden. That single line ("省略は許す、発明は許さない" / omit is fine, inventing is not) eliminated the vast majority of false positives. Classes work the same way: every documented method must exist on the real class, but unmentioned methods are treated as omitted. Testing against real-world data Not a toy README — a real, large open-source repository: Scanned 1,692 Markdown files / 4,451 code blocks. Found one genuine drift: a function documented with 2 arguments whose implementation had since moved to 1 argument. Also surfaced our own over-eager default exclusion that caused false positives — so we fixed the configuration. The real win wasn't just "it works" — it was confirming in practice that the tool does not degrade into a noisy false-positive spammer. Honest limitations It can't tell "documentation" from "illustration." A hypothetical example in a top-level README (code never meant to exist) reports as MISSING. It shines in docs/ trees whose examples mirror the codebase. Python only (for now). Other languages are counted but not checked. Name-based comparison. Default values and type annotations are ignored — it catches renames, removals, and arity/name changes, not semantic drift. Wrap-up Docs-vs-code drift only gets more dangerous as a project grows. And these "quiet but sharp-toothed" problems are exactly where deterministic tools earn their keep. This is one in a series I'm building on the same principles (deterministic / read-only / dry-run-first). A sibling, plan-drift, applies the same approach to analytics tracking plans. https://github.com/sunnydachs/doc-drift Feedback, and especially false-positive reports from your own repos, are the most useful thing you can give me — drop them in the repo issues.