大模型
arxiv.org ·
面向联邦多语言语音大语言模型的组件感知差分隐私
该论文指出，按参数量分配裁剪预算的逐层差分隐私（DP）方法在语音大语言模型（speech-LLM）上失效：当声学编码器与语言解码器的更新范数相差一个数量级时，单池逐层方法会出现跨组件预算崩溃，使词错误率（WER）显著恶化甚至训练崩溃；范数失衡较轻微时，自适应单池方法可部分恢复，说明崩溃严重程度与组件间范数比相关。作者在六种逐层方法和三种语音 LLM 架构上诊断了根因，并提出 α-split 双池分配，将编码器与 LLM 参数归一化到独立池中，同时保持联合 ℓ2 敏感度与原 (ε,δ)-DP 保证不变。在按架构校准的 α 下，该方法相比平坦 DP 恢复了 WER 效用，并使编码器针对基于说话人音色的梯度反演攻击获得 4.47 倍更紧的逐组件噪声保护，LLM 噪声开销仅增加 2.6%。
AI推荐理由 arXiv 正式提交的 cs.CL 论文，摘要给出明确问题诊断（语音 LLM 中声学编码器与语言解码器更新范数相差一个数量级导致单池 per-layer DP 的跨组件预算崩溃）与可验证方法（α-split 双池分配，保持联合 ℓ2 敏感度与 (ε,δ)-DP 保证不变），并报告在架构校准 α 下恢复 WER 效用、编码器获得 4.47× 更紧的逐组件噪声保护、LLM 噪声开销仅 +2.6%。新信息具体、用途明确（联邦多语言语音 LLM 的隐私-效用权衡），局限是仅摘要级证据、未提供代码或复现细节，且为预印本未经同行评审。
72
正文
Computer Science > Computation and Language
arXiv:2609.11762 (cs)
[Submitted on 10 Sep 2026]
Title:Component-Aware Differential Privacy for Federated Multilingual Speech-LLMs
Authors:Jordi Luque, Fernando López, Aleix Sant
View a PDF of the paper titled Component-Aware Differential Privacy for Federated Multilingual Speech-LLMs, by Jordi Luque and 2 other authors
View PDF
HTML (experimental)
Abstract:Per-layer differential privacy (DP) clipping improves gradient fidelity in federated learning by allocating per-matrix clipping budgets proportional to parameter count. We show that this recipe breaks for speech large language models (speech-LLMs), when the acoustic encoder and the language decoder differ by an order of magnitude in update norm. Single-pool per-layer methods suffer \emph{cross-component budget collapse}, dragging word error rate (WER) far from flat global clipping or collapsing training entirely. When the norm imbalance is milder, adaptive single-pool methods partially recover, confirming that collapse severity scales with the inter-component norm ratio. We empirically diagnose the root cause across six per-layer methods and three speech-LLM architectures. We then propose \emph{$\alpha$-split}, a two-pool allocation that normalises encoder and LLM parameters into independent pools, and show that joint $\ell_2$ sensitivity and the original $(\varepsilon,\delta)$-DP guarantee are unchanged. At architecture-calibrated $\alpha$, our method recovers WER utility compared to flat DP, while granting the encoder $4.47{\times}$ tighter per-component noise protection against speaker voice-based gradient-inversion attacks at only $+2.6\%$ LLM noise overhead.
Comments:
Subjects:
Computation and Language (cs.CL)
Cite as:
arXiv:2609.11762 [cs.CL]
(or
arXiv:2609.11762v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.11762
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Jordi Luque [view email]
[v1]
Thu, 10 Sep 2026 16:17:19 UTC (1,003 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Component-Aware Differential Privacy for Federated Multilingual Speech-LLMs, by Jordi Luque and 2 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)