大模型
arxiv.org ·
LLM 与语音：集成 vs. 组合
该论文比较了将大语言模型用于自动语音识别的两种路线：声学模型与 LLM 的紧耦合（speech LLM）与通过 shallow fusion 的传统组合方式，并对标签单元、LLM 规模、注意力接口、编码器下采样和长度归一化做了消融。在 LibriSpeech 与 Loquacious 域内数据上，shallow fusion 在各模型规模下均稳定优于紧耦合；但在更异构的 HuggingFace ASR leaderboard 上，集成式 prefix LLM 的平均 WER 低于 shallow fusion，增益主要来自域外语料。作者还研究了与 CTC 模型联合识别以缓解语音 LLM 幻觉，并给出相应优化。
AI推荐理由 arXiv 论文（v4，2026-09-18 修订），作者含 Hermann Ney 等，摘要给出明确实验结论：在 LibriSpeech/Loquacious 域内数据上 shallow fusion 一致优于 AM 与 LLM 的紧耦合，而在 HuggingFace ASR leaderboard 这类异构数据上集成式 prefix LLM 平均 WER 更低、增益集中在域外语料；还涉及 CTC 联合识别缓解语音 LLM 幻觉。属可验证的对比研究，对语音 LLM 评测基线选择有实际参考价值，但结论限于 ASR 场景，未提供代码或数据链接。
72
正文
Electrical Engineering and Systems Science > Audio and Speech Processing
arXiv:2603.15045 (eess)
[Submitted on 16 Mar 2026 (v1), last revised 18 Sep 2026 (this version, v4)]
Title:LLMs and Speech: Integration vs. Combination
Authors:Robin Schmitt, Albert Zeyer, Mohammad Zeineldeen, Ralf Schlüter, Hermann Ney
View a PDF of the paper titled LLMs and Speech: Integration vs. Combination, by Robin Schmitt and 4 other authors
View PDF
HTML (experimental)
Abstract:In this work, we study different approaches to utilize large language models (LLMs) for automatic speech recognition (ASR). Specifically, we compare the tight integration of an acoustic model (AM) with the LLM ("speech LLM") to the traditional way of combining AM and LLM via shallow fusion and provide ablations on the effect of different label units and LLM sizes. For tight integration, we further examine the effect of attention interfaces, encoder downsampling, and length normalization. Furthermore, we investigate joint recognition with a CTC model to mitigate hallucinations of speech LLMs and present effective optimizations. We train and evaluate on LibriSpeech and Loquacious and additionally evaluate on the HuggingFace ASR leaderboard. Across model sizes, we find that shallow fusion consistently outperforms tight integration of AM and LLM on in-domain data, highlighting the importance of strong shallow-fusion baselines when evaluating speech LLMs for ASR. On the more heterogeneous HuggingFace ASR leaderboard, however, the integrated prefix LLM achieves lower average WER than shallow fusion, with gains concentrated on out-of-domain corpora.
Comments:
Subjects:
Audio and Speech Processing (eess.AS)
Cite as:
arXiv:2603.15045 [eess.AS]
(or
arXiv:2603.15045v4 [eess.AS] for this version)
https://doi.org/10.48550/arXiv.2603.15045
Focus to learn more
arXiv-issued DOI via DataCite
Submission history
From: Robin Schmitt [view email]
[v1]
Mon, 16 Mar 2026 09:57:06 UTC (84 KB)
[v2]
Thu, 16 Apr 2026 08:18:32 UTC (84 KB)
[v3]
Tue, 7 Jul 2026 07:07:08 UTC (131 KB)
[v4]
Fri, 18 Sep 2026 11:55:01 UTC (132 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled LLMs and Speech: Integration vs. Combination, by Robin Schmitt and 4 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
eess.AS
new
|
recent
| 2026-03
Change to browse by:
eess
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)