大模型
arxiv.org ·
可解码但被错误路由：稀疏特征揭示视觉语言模型在有害模因检测中的读出差距
该 arXiv 预印本用稀疏自编码器、角色条件探针、因果干预和恢复实验，在 Gemma-3 与 Qwen3.5 上区分有害模因误分类是源于内部证据缺失还是证据无法路由到输出。在六个有害内容基准及西班牙语、印地语-英语混合语料上，稀疏读出均优于原生预测：Qwen 平均宏 F1 从 0.432 升至 0.740，Gemma 从 0.532 升至 0.714；仅校准路由可恢复平均差距的 93.3%，探针蒸馏 LoRA 能改善原生预测，但多任务共享适配出现负迁移。Gemma-3-12B 在 Facebook Hateful Memes 上呈现分布式 rank-32 图文交互，宏 F1 达 0.756 对原生 0.685。作者认为路由而非表征本身是有害模因分类的反复出现的瓶颈。
AI推荐理由 arXiv 预印本，作者与摘要信息完整，实验设计（稀疏自编码器、角色条件探针、因果干预、恢复实验）与多基准、多语言评估描述具体，事实明确度高。核心新信息是区分视觉语言模型有害模因检测失败源于“内部证据缺失”还是“证据无法路由到输出”，并给出 Qwen 稀疏读出 0.740 对原生 0.432、Gemma 0.532 提升至 0.714、仅校准路由恢复 93.3% 平均差距等可验证数字，对理解 VLM 可解释性与读出瓶颈有实际价值。局限在于均为预印本结果，未经过同行评审，且任务局限于有害模因分类，泛化性待验证。
78
正文
Computer Science > Computer Vision and Pattern Recognition
arXiv:2609.18860 (cs)
[Submitted on 16 Sep 2026]
Title:Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detection
Authors:Girish A. Koushik, Diptesh Kanojia, Helen Treharne
View a PDF of the paper titled Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detection, by Girish A. Koushik and 2 other authors
View PDF
HTML (experimental)
Abstract:When a large vision-language model misclassifies a harmful meme, the failure may reflect missing internal evidence or an inability to route represented evidence to its output. We distinguish these cases in Gemma-3 and Qwen3.5 using sparse autoencoders, role-conditioned probes, causal interventions, and recovery experiments across six harmful content benchmarks, with additional Spanish and Hindi-English code-mixed evaluations. Sparse readouts outperform native prediction on all six primary binary tasks: Qwen averages $0.740$ versus $0.432$ native macro-F1, while residual reconstruction reaches $0.486$, whereas Gemma improves from $0.532$ to $0.714$. These differences reflect supervised accessibility rather than a pre-existing, native decision rule, and the most influential token role depends on the task. Under the evaluated score scales, Qwen silent-feature ablation is $24-63$ times more probe-sensitive, whereas routed-feature patching on literal yes/no tasks is $16-140$ times more output-sensitive. Calibration-only routing recovers $93.3$% of the mean gap, and probe-distilled LoRA improves native predictions, although shared multi-task adaptation causes negative transfer. A case study of Gemma-3-12B on Facebook Hateful Memes finds a distributed rank-32 image-prompt interaction, reaching $0.756$ versus $0.685$ native macro-F1. Robustness controls show that the signal extends beyond English, is not explained solely by accompanying OCR, and depends on paired visual evidence. Thus, routing, rather than representation alone, is a recurring bottleneck in harmful meme classification.
Comments:
Subjects:
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as:
arXiv:2609.18860 [cs.CV]
(or
arXiv:2609.18860v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2609.18860
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Girish A. Koushik [view email]
[v1]
Wed, 16 Sep 2026 16:00:36 UTC (7,061 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detection, by Girish A. Koushik and 2 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CV
new
|
recent
| 2026-09
Change to browse by:
cs
cs.AI
cs.CL
cs.LG
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)