大模型
arxiv.org ·
面向大语言模型推理的自对弈搜索蒸馏
论文提出 Self-Play Search Distillation（SPSD）框架，利用在棋盘游戏上训练的类 MuZero 网络进行自对弈，将搜索过程转化为包含偏好决策、备选方案、对手回应和价值估计的结构化推理数据，再蒸馏为超人类思维链用于 LLM 监督训练。在 Qwen3-4B-Base 上，六个数学基准平均分从 24.1 升至 36.6，held-out 游戏胜率从 15% 升至 45%，显示仅用自对弈搜索记录训练也能迁移到未见数学任务，提供了一种低标注成本的合成数据生成路径。
AI推荐理由 arXiv 新论文，提出 Self-Play Search Distillation（SPSD）框架：用类似 MuZero 的棋盘游戏自对弈网络生成超人类合成数据，将搜索记录转为带环境监督的思维链，用于训练 LLM 推理。摘要给出具体量化结果：Qwen3-4B-Base 在六个数学基准上的均值从 24.1 提升到 36.6，held-out 游戏胜率从 15% 提升到 45%，且仅用自对弈搜索记录训练即可迁移到未见数学任务。新信息明确、可验证，属于方法+实验型研究，对合成数据与推理训练方向有参考价值；局限是仅摘要可见，未提供代码、数据或复现细节，效果需进一步验证。
78
正文
Computer Science > Artificial Intelligence
arXiv:2609.30936 (cs)
[Submitted on 25 Sep 2026]
Title:Self-Play Search Distillation for Large Language Model Reasoning
Authors:Lorenzo Molfetta, Wai-Chung Kwan, Giacomo Frisoni, Luca Ragazzi, Gianluca Moro, Pavlos Vougiouklis, Jeff Z. Pan, Pasquale Minervini
View a PDF of the paper titled Self-Play Search Distillation for Large Language Model Reasoning, by Lorenzo Molfetta and 7 other authors
View PDF
Abstract:Improving reasoning abilities in Large Language Models (LLMs) requires high-quality data that exposes difficult decisions, competing alternatives, and their consequences. Data scarcity is driven by the low quality of synthetic data and the cost of human labeling. We introduce Self-Play Search Distillation (SPSD), a framework for generating superhuman synthetic data via self-play of MuZero-like networks trained on board games. SPSD uses executable environments to turn search into structured reasoning problems. At each state, the expert identifies a preferred decision, plausible alternatives, plausible opponent replies, and value estimates. By converting the self-play search records into superhuman chains-of-thought, we train LLMs with environment-grounded supervision. Although trained only on self-play search records, SPSD transfers to unseen mathematics. On Qwen3-4B-Base, it raises the mean over six mathematics benchmarks from 24.1 to 36.6 while increasing the held-out-game win rate from 15% to 45%. SPSD offers an annotation-efficient way to create high-quality synthetic data for improving LLM performance in reasoning tasks.
Subjects:
Artificial Intelligence (cs.AI)
Cite as:
arXiv:2609.30936 [cs.AI]
(or
arXiv:2609.30936v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.30936
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Lorenzo Molfetta [view email]
[v1]
Fri, 25 Sep 2026 07:53:21 UTC (2,137 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Self-Play Search Distillation for Large Language Model Reasoning, by Lorenzo Molfetta and 7 other authors
View PDF
TeX Source
view license
Current browse context:
cs.AI
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)