大模型
arxiv.org ·
在检索与硬件约束下评估面向土耳其语领域文档的开放权重 LLM
该论文针对土耳其语长篇幅、结构复杂的领域文档问答，在资源受限的本地部署条件下评测 5 个 7B-8B 开放权重模型。主基准为 109 页工业研发报告衍生的 100 道经系统验证的问题，并用 112 页公共部门报告及独立构建的 100 题集复现评测协议；全部模型在 6GB VRAM 的 NVIDIA RTX 3050 笔记本 GPU 上以受控提示、解码和 4-bit 量化运行。方法学贡献是提出证据标注评测协议，无需额外模型调用即可区分检索失败与下游推理失败。主基准端到端准确率为 49%-75%；对 7 种词法、稠密与混合检索配置使用 95% Wilson 区间和精确配对 McNemar 检验比较，均未在两个文档上显著优于字符 TF-IDF 基线。证据召回率在两份报告上饱和方式不同，说明检索与有效上下文容量对部分文档构成约束、对另一些则不然。
AI推荐理由 arXiv 预印本，方法描述具体：在 6GB VRAM 的 RTX 3050 上以 4-bit 量化本地评测 5 个 7B-8B 开放权重模型，主基准为 109 页工业研发报告衍生的 100 题，并用 112 页公共部门报告与独立 100 题复现；提出可区分检索失败与推理失败的证据标注协议，端到端准确率 49%-75%，7 种检索配置均未显著优于字符 TF-IDF 基线。局限是仅限土耳其语领域文档、样本规模有限，结论不宜外推到其他语言或更大模型。
72
正文
Computer Science > Computation and Language
arXiv:2609.28007 (cs)
[Submitted on 23 Sep 2026]
Title:Evaluating Open-Weight LLMs for Turkish Domain Documents Under Retrieval and Hardware Constraints
Authors:Imtiaz Ul Hassan, Öykü Akbulut, Onur Kaya, Ardhendu Behera, Swagat Kumar, Peter Matthew, Yonghuai Liu
View a PDF of the paper titled Evaluating Open-Weight LLMs for Turkish Domain Documents Under Retrieval and Hardware Constraints, by Imtiaz Ul Hassan and 6 other authors
View PDF
HTML (experimental)
Abstract:Most Turkish-capable large language models (LLMs) are evaluated using general-purpose benchmarks rather than long, structurally complex domain documents. This paper evaluates five open-weight 7B-8B models for Turkish document question answering under a resource-constrained local deployment setting. The primary benchmark contains 100 systematically validated questions derived from a 109-page industrial R&D report, and the evaluation protocol is replicated using a second 112-page public-sector report and an independently constructed 100-question set. All models are evaluated locally on an NVIDIA RTX 3050 laptop GPU with 6 GB VRAM using controlled prompting, decoding, and 4-bit quantisation.
The principal methodological contribution is an evidence-annotated evaluation protocol that separates retrieval failure from downstream model reasoning failure without requiring additional model calls. On the primary benchmark, end-to-end accuracy ranges from 49% to 75%. Seven lexical, dense, and hybrid retrieval configurations are additionally compared using 95% Wilson intervals and exact paired McNemar tests; none significantly outperforms the character TF-IDF baseline on either document. Evidence recall saturates differently across the two reports, showing that retrieval and effective context capacity can be binding constraints for some documents but not others. These results demonstrate that model selection, retrieval behaviour, and hardware limits must be evaluated separately when deploying open-weight LLMs for Turkish domain documents.
Comments:
Subjects:
Computation and Language (cs.CL); Information Retrieval (cs.IR); Machine Learning (cs.LG)
Cite as:
arXiv:2609.28007 [cs.CL]
(or
arXiv:2609.28007v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.28007
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Imtiaz UL Hassan [view email]
[v1]
Wed, 23 Sep 2026 12:33:57 UTC (1,696 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Evaluating Open-Weight LLMs for Turkish Domain Documents Under Retrieval and Hardware Constraints, by Imtiaz Ul Hassan and 6 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-09
Change to browse by:
cs
cs.IR
cs.LG
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)