大模型
arxiv.org ·
PonderPounce：将预训练多模态大模型作为机器人控制的回合上下文引擎
该论文提出 PonderPounce，不另建专用记忆模块，而是复用预训练 MLLM 的原生因果上下文作为回合记忆：System 2 的 PONDER 整合回合历史与示范生成连续认知，System 1 的 POUNCE 异步以最新认知及其时效为条件执行控制，两者端到端联合训练、无需单独桥接预训练。在 H100 上单次推理 p50 延迟为认知刷新 78ms、动作模型调用 25ms。RoboMME 上基础数据规模成功率 60.83%，9 倍数据达 75.54%，对比 FrameSamp+Modul 的 44.51% 与 57.88%；基础规模下 PONDER 由 0.8B 扩至 9B 提升 6.71 个百分点；无执行历史的 9B PONDER 在同等监督下仅 26.21%。RoboCasa-DC 成功率 12.5%，真机四任务异步执行平均成功率 60.98%，对比 40.67%。
AI推荐理由 arXiv 论文（v2，2026-09-17 修订），摘要给出明确方法（复用 MLLM 原生因果上下文作为 episode memory，PONDER 认知模型 + POUNCE 动作模型异步耦合、端到端联合训练）与可核验的量化结果：RoboMME 基础数据规模成功率 60.83%、9 倍数据 75.54%，对比 FrameSamp+Modul 的 44.51%/57.88%；H100 上 p50 延迟 78ms（认知刷新）与 25ms（动作调用）；PONDER 从 0.8B 扩到 9B 提升 6.71 个百分点；无执行历史的 9B PONDER 仅 26.21%；RoboCasa-DC 12.5%，真机四任务平均 60.98% 对 40.67%。信息密度高、有对照与消融，属机器人+多模态模型交叉方向的可复现研究。局限：仅摘要级证据，未提供代码/权重与统计显著性，真机任务规模有限，且论文为预印本未经同行评审，效果需独立复现验证。
78
正文
Computer Science > Robotics
arXiv:2608.24115 (cs)
[Submitted on 25 Aug 2026 (v1), last revised 17 Sep 2026 (this version, v2)]
Title:PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Control
Authors:Suhwan Choi, Jaeyoon Jung, Sungkyung Kim, Yunsung Lee, Youngjae Yu
View a PDF of the paper titled PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Control, by Suhwan Choi and 4 other authors
View PDF
HTML (experimental)
Abstract:Multimodal large language models (MLLMs) can integrate long visual histories and infer behavior from a few examples, yet vision-language-action models rarely use this capacity as episode memory. Instead of a purpose-built memory module, PONDERPOUNCE reuses an MLLM's native causal context. PONDER, a pretrained System 2 MLLM, integrates episode history and demonstrations to produce continuous cognition. POUNCE, a System 1 action model, asynchronously conditions control on the newest cognition and its age. Both are jointly trained end to end without separate bridge pretraining. Optimized per-call inference on an H100 achieves p50 latencies of 78 ms for cognition-only refresh and 25 ms for action-model invocation. On RoboMME, PONDERPOUNCE achieves 60.83% success at the base data scale and 75.54% with 9x data, compared with 44.51% and 57.88% for FrameSamp+Modul. At base scale, scaling PONDER from 0.8B to 9B adds 6.71 percentage points with the POUNCE architecture unchanged. A separately trained 9B PONDER without execution history achieves only 26.21% under matched supervision. PONDERPOUNCE also achieves 12.5% success on RoboCasa-DC and demonstrates real-world applicability on four tasks under asynchronous execution, with 60.98% mean success versus 40.67% for FrameSamp+Modul.
Comments:
Subjects:
Robotics (cs.RO); Artificial Intelligence (cs.AI)
Cite as:
arXiv:2608.24115 [cs.RO]
(or
arXiv:2608.24115v2 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2608.24115
Focus to learn more
arXiv-issued DOI via DataCite
Submission history
From: Jaeyoon Jung [view email]
[v1]
Tue, 25 Aug 2026 06:24:36 UTC (798 KB)
[v2]
Thu, 17 Sep 2026 08:39:26 UTC (1,061 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Control, by Suhwan Choi and 4 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.RO
new
|
recent
| 2026-08
Change to browse by:
cs
cs.AI
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)