大模型
arxiv.org ·
BASIS：面向 LLM 推理的批量优势估计，通过单次 rollout 信息共享降低价值估计误差
该论文提出 BASIS，一种无 critic 的后训练算法，用于缓解 LLM 推理强化学习（可验证奖励）中计算效率与样本效率的权衡。每个在线训练步仅对每个 prompt 采样一次 rollout，但利用整个 batch 内跨 prompt 的信息改进价值函数估计。实验显示，相比单 rollout 基线 REINFORCE++，价值估计 MSE 降低 69%，且单 rollout 的 MSE 低于使用 8 次 rollout 的组均值估计；在显著更少训练时间下，性能接近多 rollout 的 GRPO 类基线，并常优于单 rollout 的 REINFORCE 类基线。
AI推荐理由 arXiv 正式论文（v2，2026-09-15 修订），摘要给出明确方法（critic-free、每 prompt 单次 rollout、跨 batch 信息共享）与可核验的量化结果（价值函数 MSE 较 REINFORCE++ 降低 69%，单 rollout 的 MSE 低于 8 rollout 的组均值估计）。属于 LLM 推理 RL 后训练的效率改进，对训练算力与样本效率有实际参考价值；局限是仅摘要级证据，未提供基准任务、模型规模与完整实验细节，效果需看全文验证。
78
正文
Computer Science > Machine Learning
arXiv:2605.27293 (cs)
[Submitted on 26 May 2026 (v1), last revised 15 Sep 2026 (this version, v2)]
Title:BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
Authors:Shijin Gong, Erhan Xu, Kai Ye, Giulia Livieri, Francesco Quinzan, Chengchun Shi
View a PDF of the paper titled BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning, by Shijin Gong and 4 other authors
View PDF
HTML (experimental)
Abstract:Reinforcement learning with verifiable rewards has become a standard recipe for improving the reasoning abilities of large language models. Existing algorithms face a tradeoff between computational efficiency and sample efficiency in value estimation and policy learning. We introduce BASIS, a critic-free post-training algorithm designed to address this tradeoff. At each online training step, BASIS samples only one rollout per prompt, but leverages rich information across prompts in the entire batch to improve value function estimation. Our experiments demonstrate that BASIS reduces MSE in value function estimation by 69% compared to REINFORCE++, a representative single-rollout baseline, and achieves lower MSE with one rollout than group mean estimators with 8 rollouts. This improvement in value estimation translates to better policy optimization: using substantially less training time, BASIS achieves performance close to multi-rollout GRPO-type baselines and often outperforms single-rollout REINFORCE-type baselines.
Comments:
Subjects:
Machine Learning (cs.LG); Machine Learning (stat.ML)
Cite as:
arXiv:2605.27293 [cs.LG]
(or
arXiv:2605.27293v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.27293
Focus to learn more
arXiv-issued DOI via DataCite
Submission history
From: Shijin Gong [view email]
[v1]
Tue, 26 May 2026 17:06:41 UTC (3,045 KB)
[v2]
Tue, 15 Sep 2026 11:01:41 UTC (3,390 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning, by Shijin Gong and 4 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.LG
new
|
recent
| 2026-05
Change to browse by:
cs
stat
stat.ML
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
IArxiv recommender toggle
IArxiv Recommender
(What is IArxiv?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)