智能体
arxiv.org ·
不要遮蔽环境：观测监督改变智能体在强化学习下的探索方式
该 arXiv 论文提出 ActObs，在标准 SFT 仅对智能体动作 token 计算损失的基础上，额外监督轨迹中已存在的环境观测 token，且不增加数据、参数、序列 token 或前向计算。作者报告：在 Qwen3-4B 上，由 ActObs 出发的 GRPO 在 Terminal-Bench 2.0 各采样预算下 pass@k 均高于仅动作监督版本；在 Qwen3-8B 上以部分 pass@1 可靠性换取更高 pass@k（pass@16 提升 3.4pp）并解决更多不同任务；在 SFT 与 RL 均未见的 aider-polyglot 跨域代码编辑任务上，4B 模型 pass@1 提升 4.2pp。分析显示动作与观测梯度在 SFT 中迅速正交，仅动作训练会残留较大观测梯度并使环境预测低于基座模型，联合监督可避免这种单侧特化。
AI推荐理由 arXiv 新论文（2026-09-17 提交），提出 ActObs：在 SFT 阶段除动作 token 外也监督轨迹中已有的环境观测 token，不增加数据、参数、序列长度或前向次数。摘要给出具体实验：Qwen3-4B 上 GRPO 后 Terminal-Bench 2.0 各采样预算 pass@k 均高于仅动作监督；Qwen3-8B 上 pass@16 提升 3.4pp 但牺牲部分 pass@1；跨域 aider-polyglot 代码编辑在 4B 上 pass@1 提升 4.2pp。并给出机制分析：动作与观测梯度迅速正交，仅动作训练残留较大观测梯度并削弱环境预测。信息密度高、有可验证指标，对 Agent 强化学习训练流程有直接参考价值；局限是仅摘要级证据、未提供完整实验细节与复现条件，且为预印本未经同行评审。
78
正文
Computer Science > Machine Learning
arXiv:2609.20715 (cs)
[Submitted on 17 Sep 2026]
Title:Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL
Authors:Juzheng Zhang, Disha Makhija, Manoj Ghuhan Arivazhagan, Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah
View a PDF of the paper titled Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL, by Juzheng Zhang and 4 other authors
View PDF
HTML (experimental)
Abstract:Agent trajectories record what an agent does and what happens next. Yet standard supervised fine-tuning (SFT) applies loss only to agent-authored action tokens, using environment observations as context but not as prediction targets. We ask whether this convention provides the best initialization for subsequent reinforcement learning. We introduce ActObs, which also supervises the observation tokens already present in each trajectory. Although deployed agents never generate observations, learning to predict them encourages the policy to model action consequences without adding data, parameters, sequence tokens, or forward passes. The methods perform similarly after SFT but diverge after GRPO. On Qwen3-4B, GRPO from ActObs achieves higher pass@k at every evaluated sampling budget than its action-only counterpart on Terminal-Bench 2.0. On Qwen3-8B, it trades some pass@1 reliability for higher pass@k (+3.4 pp at pass@16) and solves more distinct tasks. The advantage extends to cross-domain code editing on aider-polyglot (+4.2 pp at pass@1 at 4B), whose tasks are unseen during SFT and RL. ActObs retains more entropy during RL while requiring less policy movement, leaving the final policy closer to its SFT initialization. Our analysis traces this difference to SFT: action and observation gradients rapidly become orthogonal, while action-only training leaves a large residual observation gradient and degrades environment prediction below the base model. Joint supervision prevents this one-sided specialization, preserving consequence prediction and preparing the policy for downstream exploration.
Comments:
Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as:
arXiv:2609.20715 [cs.LG]
(or
arXiv:2609.20715v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.20715
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Juzheng Zhang [view email]
[v1]
Thu, 17 Sep 2026 17:11:29 UTC (334 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL, by Juzheng Zhang and 4 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.LG
new
|
recent
| 2026-09
Change to browse by:
cs
cs.AI
cs.CL
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
IArxiv recommender toggle
IArxiv Recommender
(What is IArxiv?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)