大模型
arxiv.org ·
知情掩码：扩散大语言模型强化学习中的结构感知扰动
该论文研究扩散大语言模型（dLLM）的强化学习对齐问题。作者发现 dLLM rollout 中存在上游/下游结构：部分 token 被揭示后会显著改变邻近未解码位置的置信度（上游），另一些仅带来局部小变化（下游），并观察到掩码下游 token 能产生更良定的子问题，称为子问题难度不对称。据此提出 Informed Masking（IM），从去噪轨迹中以零额外推理成本导出逐 token 优先级分数，使掩码采样偏向 downstream token。IM 可即插即用，在 LLaDA-8B-Instruct 上接入三种现有 dLLM RL 方法，在数学与规划基准上取得最高 2.01%、8.68%、5.77% 的相对平均增益，并改善训练稳定性。
AI推荐理由 arXiv 新论文（2026-09-22 提交），提出 Informed Masking（IM）：从 dLLM 去噪轨迹中零额外推理成本地计算逐 token 优先级，将掩码采样偏向 downstream token，以缓解 RL 中似然代理估计的子问题难度不对称。方法为 plug-and-play，在 LLaDA-8B-Instruct 上接入三种 dLLM RL 方法，报告数学与规划基准相对平均提升最高 2.01%、8.68%、5.77%，并称训练更稳定。局限：仅摘要级证据，未提供代码或复现细节，提升幅度有限且集中于单一模型。
78
正文
Computer Science > Computation and Language
arXiv:2609.25927 (cs)
[Submitted on 22 Sep 2026]
Title:Informed Masking: Structure-Aware Perturbation for Reinforcement Learning in Diffusion Large Language Models
Authors:Xiaoyi Yu, Enver Sangineto, Pei Fu, Fiorenzo Parascandolo, Wenhui Tan, Ruikang Zhang, Rita Cucchiara, Ruihua Song, Jian Luan
View a PDF of the paper titled Informed Masking: Structure-Aware Perturbation for Reinforcement Learning in Diffusion Large Language Models, by Xiaoyi Yu and 8 other authors
View PDF
HTML (experimental)
Abstract:Diffusion Large Language Models (dLLMs) have emerged as an efficient alternative to autoregressive models, yet aligning them via Reinforcement Learning (RL) requires likelihood surrogates estimated from masked reconstruction subproblems under a small Monte Carlo budget per rollout. Existing methods construct these subproblems by uniform random masking, leaving open the question of which subproblems to prioritize. We identify a systematic upstream/downstream structure in dLLM rollouts. Some tokens, when revealed, trigger large confidence changes in nearby undecoded positions; we call them upstream. Others induce only small local changes and are therefore downstream. We find masking downstream tokens yields substantially better-posed subproblems than masking upstream tokens, a phenomenon we term subproblem difficulty asymmetry. Based on the observation, we propose Informed Masking (IM), which derives a per-token priority score from the denoising trajectory at zero extra inference cost and biases mask sampling toward downstream tokens. IM is plug-and-play: when plugged into three state-of-the-art dLLM RL methods on LLaDA-8B-Instruct, it delivers up to 2.01%, 8.68%, and 5.77% relative average gains on math and planning benchmarks with improved training stability.
Comments:
Subjects:
Computation and Language (cs.CL)
Cite as:
arXiv:2609.25927 [cs.CL]
(or
arXiv:2609.25927v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.25927
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Xiaoyi Yu [view email]
[v1]
Tue, 22 Sep 2026 09:31:23 UTC (843 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Informed Masking: Structure-Aware Perturbation for Reinforcement Learning in Diffusion Large Language Models, by Xiaoyi Yu and 8 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)