大模型
arxiv.org ·
样本数量还不够：候选生成策略塑造了 LLM 测试时扩展的能耗与性能
该论文研究 LLM 测试时扩展中候选生成调度对能耗与性能的影响。作者在 500 条 GSM8K 提示上用 Phi-3-mini 和 Qwen2.5-1.5B 验证：候选数 N 从 1 增至 8 时准确率分别提升 8.4 和 18.4 个百分点。随后固定 N=8，比较 1x8、2x4、4x2、8x1 四种调度（axb 表示 a 次调用、每次 b 个候选），测量延迟、吞吐、GPU 小时与 GPU 设备能耗。在 A100 上，八次串行调用的 GPU 设备能耗是一次批量调用的 4.64–4.86 倍，P95 延迟为 5.77–6.12 倍；该模式在三个独立调度的 A100 节点及 SciQ/V100 短输出实验中同样出现。结论是候选数不足以描述多候选测试时扩展的系统成本，当候选相互独立且显存允许时，更少调用、更大批量更高效，评测应同时报告生成调度与 GPU 级系统指标。
AI推荐理由 arXiv 论文（v2，2026-09-19 修订，SC26 EESP workshop 收录），有明确实验设置与量化结果：在 500 条 GSM8K 提示上，Phi-3-mini 与 Qwen2.5-1.5B 将候选数 N 从 1 增至 8，准确率分别提升 8.4 与 18.4 个百分点；固定 N=8 比较 1x8、2x4、4x2、8x1 四种生成调度，A100 上八次串行调用的 GPU 设备能耗为单次批量调用的 4.64–4.86 倍、P95 延迟为 5.77–6.12 倍，并在三个独立调度的 A100 节点及 SciQ/V100 短输出实验中复现。新信息在于指出仅用候选数 N 无法描述测试时扩展的系统成本，建议评测同时报告生成调度与 GPU 级指标。局限是模型规模较小（1.5B/Phi-3-mini）、任务限于 GSM8K/SciQ，结论外推到大规模模型与长输出场景需谨慎。
78
正文
Computer Science > Machine Learning
arXiv:2609.19499 (cs)
[Submitted on 16 Sep 2026 (v1), last revised 19 Sep 2026 (this version, v2)]
Title:Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling
Authors:Mobina Kashaniyan, Ali Jannesari
View a PDF of the paper titled Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling, by Mobina Kashaniyan and Ali Jannesari
View PDF
HTML (experimental)
Abstract:Test-time scaling can improve large language model reasoning by generating and combining multiple candidate responses. In sampling-based methods, the inference budget is often described by the number of generated candidates, N. However, N tells us how many candidates are generated, not how they are executed. The same candidate budget can be produced in one batched generation call or split across several sequential calls with smaller batch sizes. We first study the effect of increasing N on reasoning accuracy using Phi-3-mini and Qwen2.5-1.5B on 500 GSM8K prompts. As expected, increasing N from 1 to 8 improves accuracy by 8.4 percentage points for Phi-3-mini and 18.4 points for Qwen2.5-1.5B. However, accuracy alone does not show the systems cost of using a larger candidate budget. We therefore fix N = 8 and compare four generation schedules: 1x8, 2x4, 4x2, and 8x1, where axb denotes a generation calls with b candidates per call. We measure latency, throughput, GPU-hours, and gross GPU-device energy while keeping the total candidate count fixed. On A100 GPUs, eight serial calls use 4.64-4.86x as much gross GPU-device energy and have 5.77-6.12x the P95 latency of one batched call with eight candidates. The same pattern appears across three independently scheduled A100 nodes per model and in short-output SciQ/V100 experiments. These results show that candidate count alone is not enough to describe the systems cost of multi-candidate test-time scaling. When candidates are independent and memory allows it, fewer generation calls with larger batch sizes are more efficient. Evaluations should therefore report not only candidate count and accuracy, but also generation schedule and GPU-level systems metrics.
Comments:
Subjects:
Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC); Performance (cs.PF)
Cite as:
arXiv:2609.19499 [cs.LG]
(or
arXiv:2609.19499v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.19499
Focus to learn more
arXiv-issued DOI via DataCite
Journal reference:
In Proc. of the 3rd IEEE/ACM SC26 Workshop on Energy Efficiency with Sustainable Performance (EESP), co-located with SC26, Chicago, USA,2026
Submission history
From: Mobina Kashaniyan [view email]
[v1]
Wed, 16 Sep 2026 23:29:57 UTC (1,617 KB)
[v2]
Sat, 19 Sep 2026 17:04:07 UTC (1,617 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling, by Mobina Kashaniyan and Ali Jannesari
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.LG
new
|
recent
| 2026-09
Change to browse by:
cs
cs.DC
cs.PF
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
IArxiv recommender toggle
IArxiv Recommender
(What is IArxiv?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)