给绿色构建加两把锁：面向 AI C++ 评测的 prompt 哈希与 RSS 上限

A patch landed with every assertion green. The return values matched the golden file. Then a latency dashboard moved: an inner loop that once touched a stack buffer now called std::vector::resize on every request. The eval harness never measured RSS or wall time, and it never hashed the prompt that produced the patch. That failure mode is common in AI C++ evals. Output-only golden cases miss it. This article describes a small harness that adds two locks. The first lock pins the prompt text. The second lock caps wall time and resident set size. Together they catch silent regressions that unit tests still call success. The sample below is a worked example, not a published benchmark. The hole output goldens leave Functional goldens are necessary. They are not sufficient. A model can preserve stdout, preserve exit code, and still change allocation shape, cache behavior, or exception paths. A later prompt edit can also drift the eval without anyone noticing, because the .txt expected files still match. Performance-oriented test design faces the same trap as API load tests that only check HTTP 200. A realistic gate needs a budget, a workload, and a frozen input. For C++ patches the workload is a tight binary. The frozen input is the prompt file plus the golden stdin. The budget is a contract the grader refuses to negotiate. Artifact: a budget contract The contract is a JSON file checked into the eval repo. It is not inferred from the last green run. Operators who let the model rewrite the contract are grading the model against itself. { "prompt_path": "prompts/hot_path_v3.md", "prompt_sha256": "REPLACE_WITH_SHA256", "binary": "./build/hot_path", "argv": ["--n", "200000"], "stdin_path": "goldens/hot_path.in", "stdout_sha256": "REPLACE_WITH_STDOUT_SHA", "expect_exit": 0, "max_wall_ms": 250, "max_rss_kb": 32768, "warmup_runs": 1, "timed_runs": 3 } Enter fullscreen mode Exit fullscreen mode max_wall_ms and max_rss_kb are local policy, not product claims. They must be filled from a reference binary on the same machine class that will grade patches. Mixing a laptop measurement with a remote VM measurement invalidates the cap. Step 1: freeze the prompt Prompt drift is a silent regression of a different kind. The tests stay green while the instructions change. The harness hashes the prompt file before it compiles anything. # prompt_lock.py — example grader helper from __future__ import annotations import hashlib from pathlib import Path def sha256_file(path: Path) -> str: h = hashlib.sha256() with path.open("rb") as fh: for chunk in iter(lambda: fh.read(65536), b""): h.update(chunk) return h.hexdigest() def assert_prompt_locked(contract: dict, root: Path) -> None: prompt = root / contract["prompt_path"] digest = sha256_file(prompt) expected = contract["prompt_sha256"].lower() if digest != expected: raise SystemExit( f"prompt hash mismatch: {prompt} got {digest} want {expected}" ) Enter fullscreen mode Exit fullscreen mode A prompt bump is an explicit commit. The operator updates prompt_sha256 in the same change that edits the markdown. Eval runs that skip this check are not comparable across days. Step 2: keep a boring reference binary The subject under test is a small hot path. The listing is illustrative. It is enough to show how a “correct” rewrite becomes a fatter process. // hot_path.cpp — example workload, not production code #include #include #include #include int main(int argc, char** argv) { std::uint32_t n = 1000; for (int i = 1; i + 1 (std::stoul(argv[i + 1])); } } std::uint64_t acc = 0; std::vector buf; buf.reserve(n); // reference: one allocation for (std::uint32_t i = 0; i tuple[int, str, int, float]: stdin_data = (root / contract["stdin_path"]).read_bytes() start = time.perf_counter() proc = subprocess.Popen( [str(root / contract["binary"]), *contract["argv"]], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, ) out, _err = proc.communicate(stdin_data) wall_ms = (time.perf_counter() - start) * 1000.0 usage = resource.getrusage(resource.RUSAGE_CHILDREN) rss_kb = int(usage.ru_maxrss) return proc.returncode, out.decode("utf-8", "replace"), rss_kb, wall_ms def grade(contract_path: Path) -> None: root = contract_path.parent contract = json.loads(contract_path.read_text()) assert_prompt_locked(contract, root) for _ in range(int(contract["warmup_runs"])): run_once(contract, root) walls, rsses, outputs, codes = [], [], [], [] for _ in range(int(contract["timed_runs"])): code, out, rss_kb, wall_ms = run_once(contract, root) codes.append(code) outputs.append(out) rsses.append(rss_kb) walls.append(wall_ms) stdout_digest = sha256_file # placeholder; hash the text instead text = outputs[0] import hashlib digest = hashlib.sha256(text.encode()).hexdigest() failures = [] if any(c != contract["expect_exit"] for c in codes): failures.append(f"exit {codes} want {contract['expect_exit']}") if any(o != text for o in outputs): failures.append("stdout not stable across timed runs") if digest != contract["stdout_sha256"].lower(): failures.append(f"stdout hash {digest} want {contract['stdout_sha256']}") if max(walls) > float(contract["max_wall_ms"]): failures.append(f"wall {max(walls):.1f}ms > {contract['max_wall_ms']}") if max(rsses) > int(contract["max_rss_kb"]): failures.append(f"rss {max(rsses)}KB > {contract['max_rss_kb']}") if failures: raise SystemExit("FAIL\n" + "\n".join(failures)) print( f"PASS wall_ms={max(walls):.1f} rss_kb={max(rsses)} stdout={digest[:12]}" ) if __name__ == "__main__": grade(Path("contract.json")) Enter fullscreen mode Exit fullscreen mode ru_maxrss units differ across kernels. On Linux the value is kilobytes. On some BSD systems it is bytes. The contract must document the platform. Cross-OS comparison without a unit note is a false fail or a false pass. Step 4: put the model on the other side of the contract The grader does not call a network API. It consumes a patch, builds a binary, and reads contract.json. Patches may come from a local checkout or from a remote coding session. Some teams generate those patches with MonkeyCode. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and a free server option are relevant only as a place to produce candidate diffs. They do not replace the local compiler, the prompt hash, or the RSS cap. The operator copies the same prompts/, goldens/, and contract.json onto the machine that builds. The remote side never edits the caps. If a candidate deletes buf.reserve and reallocates inside the loop, stdout can stay identical while RSS or wall time crosses the line. That is the silent regression this harness exists to name. Step 5: log enough to replay a fail A fail that cannot be replayed will be ignored. Each run should print a single machine-readable line plus the human FAIL list. python3 budget_grade.py | tee -a eval.log # expected shape on success: # PASS wall_ms=171.4 rss_kb=8124 stdout=9c1a0b7e4d21 Enter fullscreen mode Exit fullscreen mode Keep compiler version, libc, and nproc in the log header. Resource caps without that header are not evidence. They are souvenirs. What this gate does not catch The harness does not prove algorithmic complexity. A 250 ms cap on n=200000 can miss a quadratic path that only appears at production scale. It does not catch data races. It does not catch undefined behavior that happens to be fast. It does not catch a model that cheats by shrinking n in argv if the contract is not enforced. Wall time on a busy shared CPU is noisy. RSS is stabler than wall time and still not a memory-correctness proof. Operators who need allocation truth should add a custom counter or a sanitizer build as a separate job. Mixing ASan and -O2 budgets in one contract confuses both signals. Who should not use this approach This workflow does not fit interactive GUI patches, GPU kernels, or programs whose RSS is dominated by a language runtime warmup. It does not fit evals that already ship only property tests with no representative n. It does not fit teams that cannot pin a compiler. It also does not fit anyone hoping a free remote session will produce portable timing numbers. Caps are local. Prompts are hashed. Binaries are built on the grader host. A resource contract is a second lock on a green build. The first lock is the prompt digest. Neither lock is a substitute for reading the diff. They only stop the eval from celebrating a slower, fatter, still-correct patch as progress.