一个 YAML 注释截断了 AI 编码成功标准，而验证门仍然通过

When I hand implementation work to a coding agent, I want the definition of “done” to exist before the implementation does. That usually means writing explicit success criteria, keeping them in a machine-readable form, and checking them again at a verification gate. I do this in an open-source tool I maintain called spec-lane. While dogfooding it, I ran into a small YAML detail with a much larger implication: part of a success criterion was interpreted as a YAML comment before it ever reached the verification gate. The more interesting part was that the gate still passed. The bug was not in the string comparison itself. The information had already disappeared from the value being compared. It started with one line of YAML The minimal case reported in Issue #45 looked like this: success: - ledger has exactly one PhaseGate row # include the negative case too Enter fullscreen mode Exit fullscreen mode The intent described in the issue included the part after the #. But this is an unquoted YAML plain scalar. With yaml@2.9.0, it is parsed roughly as: { "success": [ "ledger has exactly one PhaseGate row" ] } Enter fullscreen mode Exit fullscreen mode So the two representations are different: source: ledger has exactly one PhaseGate row # include the negative case too parsed value: ledger has exactly one PhaseGate row Enter fullscreen mode Exit fullscreen mode The original file has not been modified. The text after # is still physically present in the YAML file. But it is a comment, so it is not part of the JavaScript value returned by yaml.parse(). That created a boundary I had not been checking closely enough: what a human sees in the source ↓ YAML parse ↓ what downstream code receives Enter fullscreen mode Exit fullscreen mode The verification gate still passed In spec-lane, the success criteria and the verification matrix are separate inputs. The success criterion lives in intent.yaml. The corresponding verification record lives in verification.yaml. For the reproduction, the matrix contained the already-truncated form: success_criteria_matrix: - criterion: ledger has exactly one PhaseGate row covered_by: test evidence: test/ledger.test.ts::records PhaseGate negation_test: no PhaseGate row fails Enter fullscreen mode Exit fullscreen mode One important caveat: those evidence and negation_test values are declarations in the reproduction fixture. For this reproduction, I did not create and execute the referenced ledger test and then prove that its behavior matched those strings. The purpose here was to reproduce the gate behavior. Reproducing the behavior before the fix I built the CLI from the revision immediately before PR #48's fix and ran the reproduction in a temporary directory. With the unquoted success criterion and the shortened matrix entry: $ lane validate ... intent.yaml is valid (phase=3_implement). Enter fullscreen mode Exit fullscreen mode Exit code: 0 Enter fullscreen mode Exit fullscreen mode Then: $ lane advance ... --phase 4_verify Advanced ...: 3_implement -> 4_verify Enter fullscreen mode Exit fullscreen mode Again: exit code: 0 Enter fullscreen mode Exit fullscreen mode The verification phase was entered. At that point, the two strings reaching the gate were effectively: success: ledger has exactly one PhaseGate row matrix criterion: ledger has exactly one PhaseGate row Enter fullscreen mode Exit fullscreen mode They matched. So the gate passed. The important distinction is this: the verification gate did not remove the text after #. That had already happened at the earlier boundary: YAML source ↓ parsed JS value Enter fullscreen mode Exit fullscreen mode The gate was comparing the values it had been given, and those values were equal. The relevant code paths are in: packages/cli/src/intent-store.ts packages/core/src/gate.ts The full reproduction and investigation are documented in Issue #45: https://github.com/shiki-yusuke/spec-lane/issues/45 Quoting the same criterion changed the result I then changed only the success criterion so the full text became an explicit YAML string: success: - "ledger has exactly one PhaseGate row # include the negative case too" Enter fullscreen mode Exit fullscreen mode Now the parsed value retains everything: success: ledger has exactly one PhaseGate row # include the negative case too matrix criterion: ledger has exactly one PhaseGate row Enter fullscreen mode Exit fullscreen mode With the same pre-fix CLI, both validate and advance now failed with exit code 3. The gate reported that there was no matrix row corresponding to the full success criterion. The phase remained 3_implement. So the control case looked like this: Success criterion Matrix validate advance Plain scalar; text after # is a comment Short value 0 0 Quoted; full value is preserved Short value 3 3 At least in this fixture, the result did not come from the gate failing to run. The value reaching the gate changed. That changed the outcome. Schema-valid does not mean source-equivalent The schema for this part of the success criteria is intentionally simple: success: z.array(z.string()).min(1) Enter fullscreen mode Exit fullscreen mode The shortened result: ledger has exactly one PhaseGate row Enter fullscreen mode Exit fullscreen mode is still a perfectly valid string. So after parsing, these two source forms can result in the same value: - ledger has exactly one PhaseGate row Enter fullscreen mode Exit fullscreen mode and: - ledger has exactly one PhaseGate row # include the negative case too Enter fullscreen mode Exit fullscreen mode If a validator only receives the parsed value, it can no longer tell whether: the author originally wrote the shorter criterion, or additional source text became a YAML comment. That distinction led me to separate several different claims that are easy to blur together: the value has the correct shape ≠ the original source expression was preserved Enter fullscreen mode Exit fullscreen mode And there are further boundaries after that: the string was preserved ≠ the string correctly represents human intent Enter fullscreen mode Exit fullscreen mode Likewise: an evidence field contains a test name ≠ that test was actually executed and proved the claim Enter fullscreen mode Exit fullscreen mode A schema validator can validate the structure it receives. It cannot automatically recover information that disappeared before that boundary. The v0.11.0 fix stops earlier PR #48 did not change the success-criteria comparison itself. Instead, the fix moved the check to the intent.yaml reading boundary. The reason is straightforward: once only the parsed value remains, there is not enough information to distinguish the two source forms. The implementation now roughly does this: Walk the YAML AST and identify unquoted PLAIN scalars. Use the scalar's source range to inspect the original text. Check whether spaces or tabs immediately after the scalar are followed by # on the same line. If such a candidate value appears in the parsed success criteria, reject the input before it reaches the gate. The implementation is here: https://github.com/shiki-yusuke/spec-lane/blob/64f2ca42c5929be0413e5f4f240897dffe56878b/packages/cli/src/intent-store.ts#L108-L164 The check deliberately does not rely only on the AST node's .comment property. One reason is that certain anchor forms can associate a comment with a different node. The implementation therefore combines AST type and range information with the original source text. With the v0.11.0 source, the same reproduction now behaves like this: Success criterion Matrix validate advance Plain scalar + inline comment Short value 2 2 Quoted full value Full value 0 0 In the rejected case, the phase remains 3_implement. If a literal # belongs in the success criterion, it can be made explicit by quoting the string: success: - "ledger has exactly one PhaseGate row # include the negative case too" Enter fullscreen mode Exit fullscreen mode The fix shipped in spec-lane v0.11.0: https://github.com/shiki-yusuke/spec-lane/releases/tag/v0.11.0 PR #48 contains the implementation and regression tests: https://github.com/shiki-yusuke/spec-lane/pull/48 This still does not mean the tool understands human intent The fix is intentionally narrower than that. It also has a known over-detection case. Consider: intent: business_goal: shared text # unrelated comment success: - "shared text" Enter fullscreen mode Exit fullscreen mode Nothing was truncated from the quoted success criterion. But the current check can still reject this document because another commented plain scalar has the same parsed value as the success criterion. The implementation does not prove that the two occurrences share the same semantic origin. That limitation is preserved in a regression test rather than hidden. So I would not describe the fix as: spec-lane now understands whether the author's full intent has been preserved. It does not. The design is closer to: if the source contains a value that could have been silently shortened before it reaches a success criterion, stop instead of guessing. That is a fail-closed choice. It trades some false positives for avoiding this known silent-truncation path. The part I now pay more attention to is the boundary This incident changed how I think about verification in AI-assisted development. It is tempting to think about the pipeline as: specification ↓ implementation ↓ verification Enter fullscreen mode Exit fullscreen mode But in practice there are more boundaries: Human expression what someone is trying to achieve ↓ Spec source what was actually written ↓ Parsed representation what the tool interpreted ↓ Verification what the gate compared ↓ Execution evidence what actually ran or was observed ↓ Acceptance why the result was accepted Enter fullscreen mode Exit fullscreen mode The failure in this post was mostly at: Spec source ↓ Parsed representation Enter fullscreen mode Exit fullscreen mode Everything after that can behave consistently and still verify less than a human reader thought had been written. The same distinction appears elsewhere. A test name written into a verification matrix is not the same thing as evidence that the test actually ran. A command exiting successfully is not necessarily the same thing as the intended property being proved. And a schema-valid specification is not necessarily the same thing as preserving every meaningful part of its source representation. Machine-readable specifications are useful. But once a workflow starts treating “the gate passed” as evidence of completion, I also want to know: What exactly reached the gate, where did that value come from, and what transformations happened before it got there? In this case, one small YAML line was enough to expose that boundary. References Issue #45 — reproduction and original report https://github.com/shiki-yusuke/spec-lane/issues/45 PR #48 — fix and regression tests https://github.com/shiki-yusuke/spec-lane/pull/48 spec-lane v0.11.0 https://github.com/shiki-yusuke/spec-lane/releases/tag/v0.11.0 AI-assisted disclosure: I used AI tools to help draft, translate, and edit this article. The reproduction, source inspection, and factual claims were checked against the linked issue, pull request, release, and the evidence collected during the investigation.