大模型
arxiv.org ·
释放全双工语音模型中的大语言模型能力
该论文提出 Listen-Write-Speak（LWS）文本优先三通道范式：单个自回归 LLM 在共享因果注意力上下文中持续听取用户音频、将可见自由文本作为主要输出，并并行生成实时语音回复，仅通过 Token Schema 实现、无需修改模型架构，并借助两阶段数据管线合成与输入时间线一致的逐秒认知标注。实验显示其在 Full-Duplex-Bench 上具备较强全双工交互能力，VoiceBench AlpacaEval 得分 4.72，写说一致性达 92.6%，并在 URO-Bench 上优于内部消融版本。作者认为可见书写可作为语音交互的一等输出通道而不牺牲实时性，代码与数据集已在项目页公开。
AI推荐理由 arXiv 论文（v2，2026-09-15 修订），提出 Listen-Write-Speak（LWS）三通道范式：单一自回归 LLM 在共享因果注意力上下文中同时听音频、写可见文本、并行语音回复，仅通过 Token Schema 实现、无需改架构，并用两阶段数据管线合成逐秒认知标注。给出 Full-Duplex-Bench、VoiceBench AlpacaEval 4.72、92.6% 写说一致性、URO-Bench 消融等具体指标，并称代码与数据集已公开。新意在于把文本提升为语音交互的一等输出通道，对实时语音 Agent 与结构化输出场景有参考价值；局限是摘要未披露基线对比细节与评测规模，指标需查原文核实。
78
正文
Computer Science > Computation and Language
arXiv:2606.07547 (cs)
[Submitted on 4 May 2026 (v1), last revised 15 Sep 2026 (this version, v2)]
Title:Liberating LLM Capabilities in Full-Duplex Speech Models
Authors:Luoyuan Zhang, Bokai Xu, Junbo Cui, Weiyue Sun, Yingjing Xu, Hanyu Liu, Yuan Yao
View a PDF of the paper titled Liberating LLM Capabilities in Full-Duplex Speech Models, by Luoyuan Zhang and 6 other authors
View PDF
HTML (experimental)
Abstract:Speech-based large language models are typically constrained to spoken replies, which limits their user-facing outputs to what can be verbalized and suppresses text-native capabilities such as code generation, structured analysis, and multi-step reasoning in realtime interaction, for tasks that require persistent, structured, and inspectable intermediate outputs. Existing work improves spoken reasoning or full-duplex turn-taking, but still treats text as a hidden intermediate state or a subordinate modality rather than a first-class output channel. We propose Listen-Write-Speak (LWS), a text-first tri-channel paradigm in which a single autoregressive LLM continuously listens to user audio, writes visible free-form text as its primary output, and speaks a realtime oral response in parallel under a shared causal attention context. This behavior is implemented entirely through a Token Schema, requiring no architectural modifications, and learned via a two-stage data pipeline that synthesizes per-second cognitive annotations consistent with the revealed input timeline. Empirically, LWS demonstrates strong full-duplex interaction on Full-Duplex-Bench, reaches 4.72 on VoiceBench AlpacaEval, achieves 92.6% writing-speaking consistency, and consistently outperforms its internal ablations on URO-Bench. These results suggest that visible writing can serve as a first-class output channel for speech interaction without sacrificing realtime responsiveness. The code and dataset are available on the project page: this https URL.
Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Sound (cs.SD)
Cite as:
arXiv:2606.07547 [cs.CL]
(or
arXiv:2606.07547v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.07547
Focus to learn more
arXiv-issued DOI via DataCite
Submission history
From: Luoyuan Zhang [view email]
[v1]
Mon, 4 May 2026 17:54:41 UTC (527 KB)
[v2]
Tue, 15 Sep 2026 13:55:00 UTC (793 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Liberating LLM Capabilities in Full-Duplex Speech Models, by Luoyuan Zhang and 6 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-06
Change to browse by:
cs
cs.AI
cs.SD
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)