不要在不同 token 预算下平均通过率

You taped two terminal panes to a Friday review. The left pane showed Agent A with eight greens; the right pane showed Agent B with six. Someone asked which one to keep. You almost said A. Then you scrolled the traces. Agent A retried until the cap cried. Agent B stopped after one compile and one test run. Same tasks. Different permission to spend. The ranking was a poster, not a measurement. A pass rate answers one question: did the tests go green. It is silent on three others. How many tokens did the run burn before that green. Did a budget cap get scored as a model failure. Were the two agents even allowed the same tool-call envelope. If you cannot answer those, you do not have a benchmark. You have a vibe with a percentage sign. This article is a protocol, not a leaderboard. You will pin a tiny task pack, declare a budget before the first call, write every attempt to a JSONL ledger, and publish three numbers together. The scripts below are a proposal you can copy. They are not a claim about any vendor eval, and they are not executed numbers from a production fleet. Why the single percentage is usually marketing Engineering reviews launder vibe-coding into a metric when the budget is a footnote. Two agents can share a task list and still live in different universes. One is allowed 40 tool calls and four repair loops. The other gets one shot and a 2k-token ceiling. Averaging their pass rates is like averaging two race times after giving one car a second tank of fuel. You need the budget in the same table as the outcome. Not in a blog paragraph. In the file you commit. The protocol below treats tokens, tool calls, and truncations as first-class results. Pass/fail is column four, not the headline. What you will actually produce Work in a single directory. Keep it boring. A pinned task pack with SHA-256 hashes. A budget envelope declared before any model call. A JSONL ledger with one object per attempt, including the dull failures. Three numbers you publish together, or you publish none of them: pass rate, truncation rate, and tokens-per-success. You can run this on a laptop. You can also run it on a free server so the cost of being honest is not why the controls get skipped. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option matter here only as a lab: you can execute the ledger without folding a billed retry policy into the ranking. Strip the product name out and the method still stands. Free is still a cap. Write that cap in the same file as the tasks. Step 1 — Pin a task pack you can hash Do not point at "the latest suite on the internet." Snapshot files. Hash them. If a fixture changes, the run id changes. That is the whole point. Proposed layout: eval_pack/ TASKS.md budget.yaml tasks/ 01_csv_summary/ prompt.md src/app.py tests/test_app.py 02_retry_header/ prompt.md src/app.py tests/test_app.py hashes.sha256 Enter fullscreen mode Exit fullscreen mode Keep the pack small on purpose. Eight to twelve tasks is enough to debug the protocol. It is not enough to crown a winner. Label it that way in TASKS.md. Each task needs three things: a prompt the agent may read, a hidden test file the agent must not edit, and a definition of done that is a command, not a feeling. # tasks/01_csv_summary/prompt.md Read src/app.py. Make summarize(path) return a dict with keys rows, empty_rows, numeric_columns. Do not edit tests/. Enter fullscreen mode Exit fullscreen mode Hash after you freeze the files: cd eval_pack find tasks -type f | sort | xargs sha256sum > hashes.sha256 sha256sum budget.yaml TASKS.md >> hashes.sha256 cat hashes.sha256 Enter fullscreen mode Exit fullscreen mode If hashes.sha256 drifts, you are not comparing agents. You are comparing days. Step 2 — Declare the budget before the first call Write the envelope first. Then refuse to start a run that does not load it. This is a control, not a comment. # budget.yaml envelope_id: "lab-2026-09-19-a" max_input_tokens_per_task: 8000 max_output_tokens_per_task: 2000 max_tool_calls_per_task: 12 max_repair_loops: 2 wall_clock_seconds_per_task: 180 hidden_tests_are_read_only: true Enter fullscreen mode Exit fullscreen mode Pick numbers that fit your lab. Do not copy these as if they were a standard. The standard is that both agents get the same envelope, and that truncation is recorded as its own outcome class. Outcome classes you will allow: pass — hidden tests exit 0 inside the envelope. fail — tests exit non-zero, budget still remaining. truncated — a cap fired before tests could pass. invalid — the agent edited a forbidden path, or the runner crashed in a way you cannot attribute. truncated is not fail. Mixing them is how a stingy envelope looks like a weak model. Step 3 — Log every attempt, including the boring ones JSONL beats a screenshot of a dashboard. One object per task attempt. No silent retries. Proposed record (label: schema proposal): { "run_id": "20260919T1400Z-lab-a", "envelope_id": "lab-2026-09-19-a", "task_id": "01_csv_summary", "agent_id": "agent-a", "pack_hash": "REPLACE_WITH_SHA256_OF_hashes.sha256", "input_tokens": 2410, "output_tokens": 880, "tool_calls": 5, "repair_loops": 1, "wall_clock_seconds": 47, "outcome": "pass", "test_exit_code": 0, "notes": "" } Enter fullscreen mode Exit fullscreen mode Append only. If a row is wrong, write a correction row. Do not silently edit history. mkdir -p runs touch runs/20260919T1400Z-lab-a.jsonl Enter fullscreen mode Exit fullscreen mode Your runner should dump token counts from the provider response, not from a guess. Tokenizers differ. That is a limitation, not an excuse to skip the column. Step 4 — Score three numbers, never one Proposal, not a published score. After the JSONL exists, compute: pass_rate = passes / tasks truncation_rate = truncated / tasks tokens_per_success = sum(input_tokens + output_tokens for passes) / max(passes, 1) If passes is zero, tokens_per_success is undefined. Print NA. Do not print infinity and call it a ranking. # score_ledger.py — proposal, unexecuted example import json import sys from collections import defaultdict REQUIRED = { "run_id", "envelope_id", "task_id", "agent_id", "pack_hash", "input_tokens", "output_tokens", "tool_calls", "repair_loops", "wall_clock_seconds", "outcome", } def load(path): rows = [] with open(path, encoding="utf-8") as handle: for line_no, line in enumerate(handle, 1): line = line.strip() if not line: continue row = json.loads(line) missing = REQUIRED - row.keys() if missing: raise SystemExit(f"line {line_no} missing {sorted(missing)}") rows.append(row) return rows def score(rows): grouped = defaultdict(list) for row in rows: grouped[(row["agent_id"], row["envelope_id"], row["pack_hash"])].append(row) reports = [] for key, items in grouped.items(): n = len(items) passes = [r for r in items if r["outcome"] == "pass"] trunc = [r for r in items if r["outcome"] == "truncated"] tokens = sum(r["input_tokens"] + r["output_tokens"] for r in passes) reports.append({ "agent_id": key[0], "envelope_id": key[1], "pack_hash": key[2], "n": n, "pass_rate": round(len(passes) / n, 3) if n else None, "truncation_rate": round(len(trunc) / n, 3) if n else None, "tokens_per_success": round(tokens / len(passes), 1) if passes else "NA", }) return reports if __name__ == "__main__": for report in score(load(sys.argv[1])): print(json.dumps(report, indent=2)) Enter fullscreen mode Exit fullscreen mode Run it: python3 score_ledger.py runs/20260919T1400Z-lab-a.jsonl Enter fullscreen mode Exit fullscreen mode Compare agents only when envelope_id and pack_hash match. If they do not match, the script still prints two blocks. You do the adult thing and refuse the average. A useful derived check: if truncation_rate differs by more than one task, you are mostly measuring the envelope, not the agent. Widen the budget and rerun both. Or keep the tight budget and say so in the first sentence of the writeup. Step 5 — Decision table: measurement vs advertisement Use this table before anyone pastes a percentage into Slack. Situation Quote a single pass rate? Publish instead Same pack_hash, same envelope_id, both agents logged No The three-number triple, plus n Different token caps, "same" tasks No Two separate triples, never averaged Hidden tests were edited by the agent No invalid count and a patch diff n under 8 tasks No The ledger, labeled as a protocol debug Truncation rate > 0.25 for one agent only No "envelope-bound" and the raw traces Latency on a shared free server No Tokens and outcomes only; drop wall-clock from the headline Homepage hero or vendor one-pager No Link the recipe, or do not cite the lab If a cell says no, the percentage is marketing even when the arithmetic is correct. Step 6 — Pairwise only inside the same envelope When you must pick a winner, do not subtract two pass rates. Count paired outcomes on the same task_id. Proposal rule: Drop any task where either agent is invalid. On the rest, score +1 for the agent that passed when the other did not. If both passed, award +1 to the lower token total only when the gap is larger than 15 percent. Otherwise call it a tie. If both truncated, call it a tie. You learned about the envelope, not the model. That 15 percent band is a knob. Write the knob in budget.yaml. Do not tune it after you see which agent you like. What this protocol refuses to claim It does not detect training-set contamination. It does not prove an agent is safe to ship. It does not turn a free server into a latency benchmark. Shared CPUs lie. Tokenizers lie in smaller ways. Your hidden tests lie if they only cover the happy path. It also does not make a 12-task pack into a scientific paper. Twelve tasks can falsify a sloppy ranking. They cannot support a superlative. Who should not use this Skip this approach if you need a homepage number this afternoon. Skip it if your "agent" is allowed network access the other agent is not. Skip it if you will not log truncated runs because they look ugly. Skip it if the real question is production SLA, not coding-agent comparison. And skip it if you planned to treat free-tier queue time as intelligence. If you want a lab that already offers free model access and a free server option, MonkeyCode is one place to run the ledger. The protocol does not depend on it. The only number worth quoting is the triple you can rebuild from JSONL and a hash file.