大模型
arxiv.org ·
通过块移除与约束二元优化实现 LLM 压缩
该论文将 LLM 的 transformer 块删除建模为约束二元优化（CBO）问题，并映射到 Ising glass 物理系统，其能量可作为下游性能的代理指标，从而对大量候选块删除配置进行高效排序，得到超越仅删除连续区域的高质量非平凡解。方法在深度压缩场景表现突出：Llama-3.3-70B-Instruct 压缩 50% 时，MMLU 相比其他 SOTA 块删除方法提升近 23 个百分点；在 Llama-3.1-8B-Instruct、Qwen3-14B（重训练前后）及 Llama-3.3-70B-Instruct 的轻压缩下与 SOTA 持平。该方法计算高效，仅需在校准数据集上对少量活跃参数做前向与反向传播，并可用启发式求解器在无法精确求解时快速获得下游表现良好的解；在块结构高度不均匀的 NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 上，删除 2 个注意力层或 3 个 MoE 层时在 AIME25 与 GPQA 上优于 SOTA。
AI推荐理由 arXiv 论文（v3，2026-09-11 修订），将 transformer 块删除形式化为约束二元优化（映射到 Ising glass），在 Llama-3.3-70B-Instruct 50% 压缩下 MMLU 相比 SOTA 块删除方法提升近 23 个百分点，轻压缩下与 SOTA 持平，并展示在 NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 上的通用性。新信息具体、可验证（基准与模型明确），对 LLM 压缩与推理成本有实际价值；局限是摘要未给出完整实验细节与复现条件，且为预印本未经同行评审。
78
正文
Computer Science > Machine Learning
arXiv:2602.00161 (cs)
[Submitted on 29 Jan 2026 (v1), last revised 11 Sep 2026 (this version, v3)]
Title:LLM Compression by Block Removal with Constrained Binary Optimization
Authors:David Jansen, Roman Rausch, Ali Hashemi, David Montero, Román Orús
View a PDF of the paper titled LLM Compression by Block Removal with Constrained Binary Optimization, by David Jansen and 4 other authors
View PDF
HTML (experimental)
Abstract:In this paper, we formulate the compression of large language models (LLMs) by optimally deleting transformer blocks (``block removal'') as a constrained binary optimization (CBO) problem that can be mapped to a physical system (Ising glass), whose energies are a strong proxy for downstream model performance. This formulation enables an efficient ranking of a large number of candidate block-removal configurations yielding many high-quality, non-trivial solutions beyond those only removing consecutive regions. Our method performs strongly in the deep compression regime, such as for 50% compression of Llama-3.3-70B-Instruct, where we achieve an almost 23 percentage point increase on the MMLU benchmark compared to other state-of-the-art (SOTA) block-removal methods. For lighter compression, it performs on par with those methods across several benchmarks for Llama-3.1-8B-Instruct, Qwen3-14B (both before and after retraining), as well as Llama-3.3-70B-Instruct. The approach is computationally efficient and requires only forward and backward passes on a calibration dataset for a few active parameters. Additionally, we demonstrate that using good heuristic solvers for the CBO problem provides solutions that perform well on downstream tasks in negligible runtime when it is unfeasible to solve the problem exactly. The method can be readily applied to any architecture. We illustrate this generality on the recent NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 model, which exhibits a highly inhomogeneous and challenging block structure, and where we outperform SOTA for AIME25 and GPQA when removing either 2 attention layers or 3 mixture-of-experts layers.
Comments:
Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Quantum Physics (quant-ph)
Cite as:
arXiv:2602.00161 [cs.LG]
(or
arXiv:2602.00161v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2602.00161
Focus to learn more
arXiv-issued DOI via DataCite
Submission history
From: David Jansen [view email]
[v1]
Thu, 29 Jan 2026 19:46:39 UTC (199 KB)
[v2]
Wed, 17 Jun 2026 12:42:15 UTC (144 KB)
[v3]
Fri, 11 Sep 2026 08:17:34 UTC (144 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled LLM Compression by Block Removal with Constrained Binary Optimization, by David Jansen and 4 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.LG
new
|
recent
| 2026-02
Change to browse by:
cs
cs.AI
cs.CL
quant-ph
References & Citations
INSPIRE HEP
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
IArxiv recommender toggle
IArxiv Recommender
(What is IArxiv?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)