大模型
arxiv.org ·
将开放权重多模态大语言模型适配于生成电子显微镜分割的点提示
该 arXiv 预印本研究开放权重多模态大语言模型（MLLM）能否根据自然语言请求输出坐标，为冻结的分割器 microSAM 自动生成点提示。作者将三个线粒体数据集的掩码转换为图像-指令-质心坐标训练样本，冻结 MLLM 主干与 microSAM，仅训练 LoRA 适配器。结果显示 Qwen3-VL 经监督微调与奖励优化后分割 AP50 达 0.736（未适配时为 0.247），自动提示生成（APG）为 0.773，另两个 MLLM 也达到或超过 APG；与 AP50 0.904 的监督质心热图检测器相比，Qwen3-VL 更接近标注点集与实例计数。两数据集训练可迁移至未见过的第三个数据集，三数据集训练可迁移至独立 EM 体数据，且在未见过的自然语言请求表述下表现稳定，坐标可被第二个分割器复用。作者称这是开放权重 MLLM 作为 EM 点生成器的首个可行性研究。
AI推荐理由 arXiv 预印本，作者为 Samia Mohinta 与 Albert Cardona，摘要给出明确实验设置与量化结果：冻结 MLLM 主干与 microSAM，仅训练 LoRA 适配器，将三个线粒体数据集的掩码转为质心坐标训练样本；Qwen3-VL 经监督微调与奖励优化后分割 AP50 从 0.247 提升至 0.736，自动提示生成（APG）达 0.773，另两个 MLLM 也达到或超过 APG；与监督质心热图检测器（AP50 0.904）相比，Qwen3-VL 在标注点集与实例计数上更接近。局限在于这是可行性研究，任务限于线粒体分割，且未超过监督检测器。新信息是用开放权重 MLLM 以自然语言请求生成点提示、并验证跨数据集与跨 EM 体数据的迁移及对请求表述变化的鲁棒性，对 AI 应用与生物医学图像分析有参考价值。
72
正文
Computer Science > Computer Vision and Pattern Recognition
arXiv:2609.14080 (cs)
[Submitted on 12 Sep 2026]
Title:Adapting Open-Weight MLLMs to Generate Point Prompts for Electron Microscopy Segmentation
Authors:Samia Mohinta, Albert Cardona
View a PDF of the paper titled Adapting Open-Weight MLLMs to Generate Point Prompts for Electron Microscopy Segmentation, by Samia Mohinta and 1 other authors
View PDF
HTML (experimental)
Abstract:Promptable models such as microSAM segment electron microscopy (EM) images from point prompts, but automation requires generating prompts without user input. We ask whether open-weight multimodal large language models (MLLMs) can generate them from natural-language requests by returning coordinates to a frozen segmenter. To that end, we convert masks from three mitochondria datasets into training examples, pairing images and instructions with centroid coordinates, then train LoRA adapters while freezing the MLLM backbone and microSAM. We find that Qwen3-VL reaches segmentation AP$_{50}$ $0.736$ after supervised fine-tuning and reward optimization, up from $0.247$ without adaptation, while automatic prompt generation (APG) achieves $0.773$. In addition, two other MLLMs improve, reaching or exceeding APG. When compared with a supervised centroid-heatmap detector that reaches AP$_{50}$ $0.904$ for this mitochondria task, Qwen3-VL more closely matches the annotated point set and instance counts. Moreover, training on two public datasets transfers to an unseen third, while training on all three transfers to an independent EM volume. Robustness tests show stable performance under unseen formulations of the natural-language request, while the coordinates can be reused by a second segmenter. To our knowledge, this is the first feasibility study of open-weight MLLMs as EM point generators, providing an inspectable, language-directed link between localization and mask decoding.
Comments:
Subjects:
Computer Vision and Pattern Recognition (cs.CV)
Cite as:
arXiv:2609.14080 [cs.CV]
(or
arXiv:2609.14080v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2609.14080
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Samia Mohinta [view email]
[v1]
Sat, 12 Sep 2026 17:52:34 UTC (2,708 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Adapting Open-Weight MLLMs to Generate Point Prompts for Electron Microscopy Segmentation, by Samia Mohinta and 1 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CV
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)