智能体
arxiv.org ·
CQ4OE：评估 LLM 辅助从能力问题生成本体论的基准
该论文提出 CQ4OE 基准，用于系统、可复现地评估基于 LLM 从能力问题（CQ）生成本体论的过程。针对现有评估任务表述不一、金标准缺乏 CQ 溯源、指标混淆词汇与结构逻辑充分性等局限，作者为每个本体构建 CQ 驱动的金标准 OWL 本体，显式关联 CQ 与所需类、属性和公理，并定义 CQ2Term（99 个 CQ 的术语级评估）与 CQ2Onto（118 个 CQ 的本体级评估，覆盖层级、属性建模和公理结构）两个任务。使用 9 个 LLM 在零样本、迭代和多智能体策略下实验，结果显示 LLM 恢复显式词汇比构建本体更可靠，在属性建模、层级构建和公理生成方面表现较弱。
AI推荐理由 arXiv 新提交的学术论文，提出 CQ4OE 基准，针对 LLM 从能力问题（CQ）生成本体论这一本体工程核心环节，指出现有评估任务表述不统一、金标准缺乏细粒度 CQ 溯源、指标混淆词汇重叠与结构逻辑充分性等问题。基准为每个本体构建 CQ 驱动的金标准 OWL 本体并显式关联 CQ 与所需类、属性、公理，定义 CQ2Term（99 个 CQ 的术语级评估）和 CQ2Onto（118 个 CQ 的本体级评估，含层级、属性建模、公理结构）两个任务，并用 9 个 LLM 在零样本、迭代、多智能体策略下实验，发现 LLM 恢复显式词汇比构建本体更可靠，尤其在属性建模、层级构建和公理生成上较弱。信息密度高、来源为 arXiv 预印本，但尚未经同行评审，实验结论为作者自述，需谨慎看待。
72
正文
Computer Science > Artificial Intelligence
arXiv:2609.26029 (cs)
[Submitted on 22 Sep 2026]
Title:CQ4OE: A benchmark for assessing LLM-assisted ontology generation from competency questions
Authors:Jiayi Li, Ziyuan Wang, Daniel Garijo, María Poveda-Villalón
View a PDF of the paper titled CQ4OE: A benchmark for assessing LLM-assisted ontology generation from competency questions, by Jiayi Li and 2 other authors
View PDF
HTML (experimental)
Abstract:Ontology generation from Competency Questions (CQs) is a central yet labor-intensive phase of Ontology Engineering. While large language models (LLMs) offer promising automation capabilities, current evaluations remain fragmented. Task formulations are heterogeneous, gold standards often lack fine-grained CQ provenance, metrics conflate lexical overlap with structural and logical adequacy, and reference ontologies are not always explicitly designed around the evaluation CQs. Here, we address these limitations with CQ4OE, a benchmark for the systematic and reproducible evaluation of LLM-based ontology generation from CQs. For each ontology in the benchmark, we build a CQ-driven gold OWL ontology with explicit provenance linking each CQ to the classes, properties, and axioms required to answer it. From this resource, we define two complementary evaluation tasks. CQ2Term supports term-level evaluation of CQ-specific class and property prediction over 99 CQs, and CQ2Onto supports ontology-level evaluation over 118 CQs, including hierarchy, property modeling, and axiom-level structure. We demonstrate CQ4OE with experiments using nine LLMs under zero-shot, iterative, and multi-agent generation strategies, showing that LLMs recover explicit vocabulary terms more reliably than creating ontologies, particularly in property modeling, hierarchy construction, and axiom generation.
Subjects:
Artificial Intelligence (cs.AI)
Cite as:
arXiv:2609.26029 [cs.AI]
(or
arXiv:2609.26029v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.26029
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Daniel Garijo [view email]
[v1]
Tue, 22 Sep 2026 11:32:44 UTC (597 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled CQ4OE: A benchmark for assessing LLM-assisted ontology generation from competency questions, by Jiayi Li and 2 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AI
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)