大模型
arxiv.org ·
面向新型 AI 辅助教育问题的教学评估预训练模型评测
该 arXiv 预印本评估传统机器学习、Transformer 与大语言模型在 Bloom 层级分类任务上的表现，重点关注 AI 生成教育问题带来的分布外（OOD）性能下降。基线结果显示 TF-IDF 机器学习模型 OOD Macro F1 仅 0.48，BERT 为 0.55，LLM 达 0.79；文本拼接将 ML 与 BERT 分别提升至 0.59 和 0.62，附加学习目标可提升特定数据集表现，模型重训练带来的改进最大。作者据此讨论预训练模型用于新型 AI 辅助教育问题的权衡及特征增强策略。
AI推荐理由 arXiv 预印本，摘要给出明确实验设置与量化结果（TF-IDF ML 在 OOD 上 Macro F1 0.48，BERT 0.55，LLM 0.79；文本拼接提升至 0.59/0.62），信息密度较高且可验证。但属小众教育评估任务，作者与机构无已知影响力，未提供代码或数据集链接，对 AI 模型/Agent/算力主线的直接价值有限，故兴趣分中等。
62
正文
Computer Science > Artificial Intelligence
arXiv:2609.27749 (cs)
[Submitted on 23 Sep 2026]
Title:Evaluation of pre-trained models for pedagogical assessment of novel AI-assisted educational questions
Authors:Michael Lawrence Castanares, Princess Ventures, Allan Tan
View a PDF of the paper titled Evaluation of pre-trained models for pedagogical assessment of novel AI-assisted educational questions, by Michael Lawrence Castanares and 2 other authors
View PDF
HTML (experimental)
Abstract:The surge in AI-assisted generation of educational materials has outpaced our capacity to validate their pedagogical quality. Automated evaluation using Bloom Classifier models is a promising approach to assess educational materials at scale. These models show high accuracy within-distribution dataset (IID Dataset). However, applying the same models to new out-of-distribution (OOD) datasets such as AI-assisted generated questions could show performance degradation. To identify robust classifiers under dataset shift, we evaluated traditional Machine Learning (ML), transformer, and Large Language models on the Bloom level classification task. We also explored feature-engineering strategies incorporating NLP metrics, appending the learning objectives as part of the input, and text splicing to stabilize OOD performance. Our baseline tests show that TFPOS-IDF ML models perform poorly on OOD (Macro F1-score 0.48) compared to BERT (0.55) and LLMs (0.79). Text splicing improved macro F1-score performance of ML and BERT models (0.59 and 0.62, respectively). Appending the learning objectives with the input increased model performance on specific dataset. Model retraining provided the largest improvement across models and datasets. Overall, these findings highlight the trade-off on the use of pre-trained models with novel AI-assisted educational questions and how strategic feature enhancements help address loss in performance.
Comments:
Subjects:
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
ACM classes:
I.2.7; K.3.1
Cite as:
arXiv:2609.27749 [cs.AI]
(or
arXiv:2609.27749v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.27749
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Allan Tan [view email]
[v1]
Wed, 23 Sep 2026 12:00:33 UTC (1,029 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Evaluation of pre-trained models for pedagogical assessment of novel AI-assisted educational questions, by Michael Lawrence Castanares and 2 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AI
new
|
recent
| 2026-09
Change to browse by:
cs
cs.CL
cs.LG
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)