智能体
dev.to ·
@Janz
预注册实验：2 个 vs 4 个 Agent 各跑六次，4 Agent 成本 2.115 倍但成功率未提升
作者在确定性公共修复贡献任务上做预注册对照：2-agent/3 步与 4-agent/5 步各重复 3 次，硬性上限为每次 150 次调用、0.50 美元、600 秒，选择模型为 deepseek-flash。六次运行均通过来源、覆盖、整数选择、算术、上限、端口关闭与归档哈希等有效性门禁。两组均只在 3 次中成功 1 次；4-agent 组平均调用 33 次、token 21,877、成本 0.0609 美元，约为 2-agent 组的 1.94 倍、2.09 倍和 2.115 倍，平均零贡献者 1.333 对 1.000。作者结论是更多 Agent 并非自动更好，但明确该结果不适用于其他任务或真实用户，且无人工评分，样本仅每组 3 次。方法包与信任层已以 MIT 协议公开。
AI推荐理由 预注册的 2-agent 与 4-agent 对照实验，六次运行全部通过有效性门禁，公开了方法包与信任层仓库，可复现；但样本仅每组 3 次、单一合成任务，作者本人也明确不推广到其他任务或真实用户，且无人工评分，因此结论限于该任务。
72
正文
Short version: I ran a preregistered 2-agent versus 4-agent comparison on a deterministic task.
Six live runs completed under hard caps. Both groups succeeded in exactly one of three repeats.
The 4-agent group cost 2.115× more per run and per complete success. More agents were operationally
viable; they were not better on this task.
The task
The scenario is a deterministic public-repair contribution task:
eight named participants;
each participant chooses how many repair units to contribute;
the task succeeds only if the total reaches a fixed threshold;
participant coverage is forced, so every selected agent acts;
no LLM reasoning is used for the choice; the model returns a strict JSON choice.
The point of forcing coverage was to avoid the earlier failure mode where one agent dominated every
turn and the other participants never acted.
Protocol
Item
Value
Groups
2 agents / 3 steps; 4 agents / 5 steps
Repeats
3 per group
Caps per run
150 calls / USD 0.50 / 600s
Choice model
deepseek-flash on a Responses API contract
Outcome
machine-decidable outcome.json
Validity
provenance, coverage, integer choices, arithmetic, caps, port closure, archive hashes
All six runs passed every validity gate. No parser failure, no budget breach, no port leak.
Results
Run
Agents
Total / threshold
Success
Zero contributors
Calls
Total tokens
Cost USD
2-agent r1
2
2 / 3
no
1
17
10,060
0.0269116
2-agent r2
2
3 / 3
yes
0
17
11,070
0.0313716
2-agent r3
2
0 / 3
no
2
17
10,255
0.0281116
4-agent r1
4
3 / 5
no
2
33
21,493
0.0597772
4-agent r2
4
2 / 5
no
2
33
22,679
0.0641372
4-agent r3
4
5 / 5
yes
0
33
21,459
0.0588042
Group aggregates:
Metric
2-agent
4-agent
Ratio
complete successes
1 / 3
1 / 3
1.000
mean calls
17.00
33.00
1.941
mean total tokens
10,461.67
21,877.00
2.091
mean cost USD
0.028798
0.060906
2.115
mean zero contributors
1.000
1.333
1.333
What I take from it
More agents are not automatically better. In this task they doubled cost and left more participants
contributing nothing on average.
The result is a valid negative result, not an apparatus failure. Every run completed and every
artifact passed its gates.
The 4-agent group did produce one complete success, so the mechanism is not broken. It is simply
not worth 2.115× the cost on this task.
What this does not show
It does not show that four agents are worse in general.
It does not transfer to another task or to a real user.
It says nothing about agent quality under human review, because no human scored these runs.
The sample is three repeats per group on one synthetic task.
Reproduce it
Two pieces are now public:
Method package: https://github.com/janzong/agent-lab-method (MIT, commit 6cc70156facb37baf23fe5fe57dad93d43502b91) — schema, synthetic GenMentor adapter, fixtures, tests;
Trust layer: https://github.com/janzong/agent-lab-trust (MIT, release v0.1.0-rc2) — local-first run validation, hashed reports, and a synthetic deletion proof.
The full protocol and the archived runs stay private. The result is intentionally boring: a negative result with caps, hashes, and every failed run preserved.
I am also looking for three independent reproductions by non-authors. The trust layer guide expects 13 passed under both TZ=UTC and TZ=Asia/Shanghai, report_hash a841b192981fd7e7, and deletion audit_hash 4f0193abbd49a0f9. If you run it and any hash differs, that is the most useful reply I can get. If you have a task where you believe more agents should win, that is the experiment I want to run next.