智能体
dev.to ·
@VANSH ARORA
自上而下截断为何会破坏 AI Agent，以及 Even-Span 如何修复
多数 AI 编码上下文生成器按行从上到下截断长文件，若预算只允许 400 行，则 1200 行文件的后 800 行全部丢失，模型看不到底部的导出接口、路由注册和模块绑定，会误判其不存在并生成重复或损坏代码。TokenCap 在 src/pack/evenSpan.js 中实现 even-span 分布：将文件划分为均衡区间，采样结构切片并保留 AST 函数签名，以在不超 token 预算的前提下保留关键结构信息。
AI推荐理由 文章指出 AI 编码上下文生成器常见的自上而下截断缺陷：长文件只保留前 N 行，导致模型看不到底部的 module.exports、路由注册和生命周期绑定，从而生成重复或错误代码。提出 TokenCap 的 even-span 采样方法，按区间均衡采样并保留 AST 函数签名。问题描述具体、有工程价值，但方案效果缺少量化对比数据。
66
正文
Most AI coding context generators have a hidden design flaw: they truncate long files sequentially from line 1 downward until they hit a token budget limit.
If a file has 1,200 lines and your budget allows 400 lines, the model receives lines 1 to 400. Everything from line 401 to 1200 vanishes.
In production codebases, this is fatal:
Top: imports and constants
Middle: core logic and helper functions
Bottom: exported interfaces, route registrations, module.exports, and lifecycle bindings
When the agent cannot see module.exports or class registrations at the bottom, it assumes they do not exist and generates duplicate or broken code.
The Even-Span Sampling Methodology
To solve this without blowing the token budget, TokenCap implements even-span distribution in src/pack/evenSpan.js.
The algorithm divides the file into balanced intervals and samples structural slices while preserving AST function signatures:
// src/pack/evenSpan.js overview
function computeEvenSpans(lineCount, maxLines, anchorPoints) {
// Guarantees head imports, core anchor blocks, and tail exports
// are represented proportionally within the allocated budget.
}
Enter fullscreen mode
Exit fullscreen mode
Instead of:
Lines 1 - 350: Captured
Lines 351 - 1200: [TRUNCATED]
Enter fullscreen mode
Exit fullscreen mode
Even-span provides:
Lines 1 - 80: Header, configuration, types
... [140 lines folded] ...
Lines 220 - 310: Core logic and targeted symbols
... [290 lines folded] ...
Lines 600 - 680: Lifecycle handlers and bottom exports
Enter fullscreen mode
Exit fullscreen mode
Every span boundary snaps cleanly to structural declaration boundaries rather than slicing mid-statement.
Run tokencap make to inspect how your large files are budgeted.
Read more at tokencap.vansharora.app