大模型
arxiv.org ·
本地部署 LLM 的能效：面向消费级硬件的初步 GPU 功耗定量基准测试
该 arXiv 预印本在单张消费级 GPU（RTX 4060ti 16GB）上，用 Ollama 对 18 个 0.5B–7B 开源模型做硬件级能耗基准，以 2Hz 采样 GPU 功耗，评估平均/峰值功耗、每 prompt 能耗（J/prompt）、每输出 token 能耗（J/tok）与吞吐。结果显示能效不只由参数量决定，架构与量化策略同样关键：qwen2.5:0.5b 与 tinyllama:1.1b 能耗最低（0.2747、0.3234 J/tok）且吞吐 >325 tok/s，7B-Mistral 每 token 能耗最高达最省电模型的 8.6 倍；qwen3.5:0.8b(on) 因内部推理延长而单 prompt 能耗异常偏高，提示效率指标需区分 token 生成模式。作者称其为初步基准，仅限单卡与固定 prompt 集。
AI推荐理由 arXiv 预印本，方法可复现：在单张消费级 GPU（RTX 4060ti 16GB）上用 Ollama 对 18 个 0.5B–7B 开源模型做硬件级功耗基准，以 2Hz 采样 nvidia-smi，报告 J/prompt、J/tok、吞吐等指标。具体新信息包括 qwen2.5:0.5b 与 tinyllama:1.1b 能耗最低（0.2747、0.3234 J/tok，吞吐 >325 tok/s），7B-Mistral 每 token 能耗最高可达最省电模型的 8.6 倍，以及 qwen3.5:0.8b(on) 因内部推理延长导致单 prompt 能耗异常偏高。局限：作者自述为初步基准，仅单卡、固定 prompt 集，结论不宜外推；未提供完整数据表。
72
正文
Computer Science > Artificial Intelligence
arXiv:2608.00008 (cs)
[Submitted on 12 Jun 2026 (v1), last revised 25 Sep 2026 (this version, v2)]
Title:Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardware
Authors:Philipp M. Zähl, Elja Dalipaj, Anika Hennig, Timon Bayer
View a PDF of the paper titled Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardware, by Philipp M. Z\"ahl and 3 other authors
View PDF
HTML (experimental)
Abstract:The local deployment of large language models (LLMs) is gaining traction due to privacy concerns and the desire for on-premise inference. However, the energy costs on consumer hardware remain poorly characterized, as most benchmarks focus solely on accuracy. This paper presents a reproducible, hardware-level energy benchmark of 18 open-source LLMs (0.5B to 7B parameters) executed on a single consumer GPU (RTX 4060ti 16GB). Using the Ollama inference engine, GPU power draw was sampled at 2hz via nvidia-smi across a fixed prompt set. We evaluate mean/peak power, total energy per prompt (J/prompt), energy per output token (J/tok), and throughput (tok/s). Our findings suggest that factors beyond raw parameter count, including model architecture and quantization strategy, drive energy efficiency. Specifically, qwen2.5:0.5b and tinyllama:1.1b achieve the lowest energy cost (0.2747 J/tok and 0.3234 J/tok) and the highest throughput (>325 tok/s). In contrast, the 7B-Mistral model consumes up to 8.6x more energy per token than the most efficient model. Notably, qwen3.5:0.8b(on) exhibits anomalously high per-prompt energy due to extended internal reasoning, highlighting the need to distinguish between token generation modes in efficiency metrics.
Subjects:
Artificial Intelligence (cs.AI); Emerging Technologies (cs.ET); Performance (cs.PF)
Cite as:
arXiv:2608.00008 [cs.AI]
(or
arXiv:2608.00008v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2608.00008
Focus to learn more
arXiv-issued DOI via DataCite
Submission history
From: Philipp Zähl [view email]
[v1]
Fri, 12 Jun 2026 06:53:44 UTC (170 KB)
[v2]
Fri, 25 Sep 2026 13:12:53 UTC (242 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardware, by Philipp M. Z\"ahl and 3 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AI
new
|
recent
| 2026-08
Change to browse by:
cs
cs.ET
cs.PF
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)