大模型
arxiv.org ·
数据集语言结构在大语言模型文化意识中的作用
该 arXiv 论文从数据集视角研究文化对齐：对阿拉伯语、中文、日语微调数据计算语言学、语义与结构指标并分别做 PCA，得到可解释主轴（PC1 偏语义结构、PC2 偏多样性与词汇变化、PC3 偏语言特有组织），再微调 LLaMA、Mistral、DeepSeek 并在文化知识、价值观与规范基准上评测。结果显示 PCA 描述符与下游表现相关，但最强关系随模型、基准和语言变化，没有单一通用预测因子；等规模子集干预表明，在验证相关成分与方向后，PCA 引导的数据选择可改善文化表现——阿拉伯语以 PC3 信号最强，日语 High-PC1 对 LLaMA 最有效，中文结果较弱且更依赖模型，因样本量较小仍属探索性。作者认为轻量数据集描述符可用于训练前数据诊断，但有效文化适配需要语言与架构感知的选择策略。
AI推荐理由 arXiv 论文（v2，2026-09-21 修订），研究微调数据集的语言学属性与文化对齐表现的关系：对阿拉伯语、中文、日语数据集计算轻量级语言学/语义/结构指标并做 PCA，再微调 LLaMA、Mistral、DeepSeek 并在文化知识、价值观、规范基准上评测。核心新信息是 PCA 派生描述符与下游表现相关，但最强关系随模型、基准、语言变化，不存在通用预测因子；受控等规模子集干预显示 PCA 引导选择在验证方向后可提升文化表现（阿拉伯语 PC3 信号最强，日语 High-PC1 对 LLaMA 最有效），中文结果较弱且更依赖模型、因样本量小仍属探索性。局限：结论为相关性而非因果，中文部分作者自述探索性，未给出具体提升幅度。对做多语言文化适配与数据筛选的开发者有参考价值，但属学术探索，非产品级突破。
72
正文
Computer Science > Computation and Language
arXiv:2602.01161 (cs)
[Submitted on 1 Feb 2026 (v1), last revised 21 Sep 2026 (this version, v2)]
Title:The Role of Dataset Linguistic Structure in the Cultural Awareness of Large Language Models
Authors:Reem I. Masoud, Chen Feng, Shunta Asano, Saied Alshahrani, Philip Colin Treleaven, Miguel R. D. Rodrigues
View a PDF of the paper titled The Role of Dataset Linguistic Structure in the Cultural Awareness of Large Language Models, by Reem I. Masoud and 5 other authors
View PDF
HTML (experimental)
Abstract:The global deployment of large language models (LLMs) has raised concerns about cultural misalignment, yet the linguistic properties of fine-tuning datasets used for cultural adaptation remain poorly understood. We adopt a dataset-centric view of cultural alignment and investigate which properties of post-training data are associated with cultural performance, whether they can guide data selection before fine-tuning, and how their effects vary across languages and model families. We compute lightweight linguistic, semantic, and structural metrics for Arabic, Chinese, and Japanese datasets and apply principal component analysis (PCA) separately within each language. The resulting components form broadly interpretable axes: PC1 is generally dominated by semantic structure, PC2 captures diversity and lexical variation, and PC3 reflects more language-specific organization. We fine-tune LLaMA, Mistral, and DeepSeek models and evaluate them on benchmarks of cultural knowledge, values, and norms. Although the PCA-derived dataset descriptors are associated with downstream performance, the strongest relationships vary across models, benchmarks, and languages, indicating that no single component serves as a universal predictor. Controlled, size-matched subset interventions further show that PCA-guided selection can improve cultural performance when the relevant component and direction are validated against random sampling. PC3 provides the strongest signal for Arabic, while High-PC1 is most effective for Japanese, particularly for LLaMA. Chinese results are weaker and more model-specific and remain exploratory because of smaller subset sizes. Overall, our findings show that lightweight dataset descriptors can support pre-training data diagnostics, but effective cultural adaptation requires language- and architecture-aware selection rather than a universal linguistic criterion.
Subjects:
Computation and Language (cs.CL)
Cite as:
arXiv:2602.01161 [cs.CL]
(or
arXiv:2602.01161v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2602.01161
Focus to learn more
arXiv-issued DOI via DataCite
Submission history
From: Reem Masoud [view email]
[v1]
Sun, 1 Feb 2026 11:21:30 UTC (977 KB)
[v2]
Mon, 21 Sep 2026 16:57:23 UTC (10,141 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled The Role of Dataset Linguistic Structure in the Cultural Awareness of Large Language Models, by Reem I. Masoud and 5 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new
|
recent
| 2026-02
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)