大模型
arxiv.org ·
通过视觉语言模型辅助空间认知
该 arXiv 预印本提出一个端到端框架，融合 LLM、VLM 与数字孪生技术，为视障及神经多样性用户提供具备空间认知能力的导航支持。系统用普通手机摄像头采集视频，通过 SLAM3R 从单目 RGB 序列实时生成稠密 3D 点云，并用自定义后处理算法在无需预设参考点的情况下完成多视角点云对齐，进而增强 SpatialLM 生成包含建筑元素和带方向物体包围框的结构化 3D 表示。随后由本地部署的 LLM 解析 3D 上下文，输出详细场景描述以及用户与周围物体的精确距离。作者在多种视角、绕行路线和多环境的视频场景中评估，报告 3D 场景理解与物体定位表现一致，但摘要未给出量化指标与用户研究结果。
AI推荐理由 arXiv 预印本，摘要明确给出方法链路（手机摄像头视频、SLAM3R 单目 RGB 实时稠密点云、自定义后处理做多视角对齐、SpatialLM 结构化 3D 表示、本地部署 LLM 生成场景描述与距离测量），并说明在多种视频场景下评估了 3D 场景理解与物体定位的一致性。新信息在于把 VLM/LLM、数字孪生与 SLAM 结合用于视障与神经多样性人群的空间认知辅助导航，且强调本地部署 LLM。局限：仅为预印本，未提供量化指标、基线对比与用户研究细节，'一致准确'的表述缺乏具体数据支撑，实际效果尚待验证。
72
正文
Computer Science > Artificial Intelligence
arXiv:2609.12747 (cs)
[Submitted on 11 Sep 2026]
Title:Assisted Spatial Cognition Through Vision-Language Models
Authors:H. Riaz, J. B. Fernandez, I. Mills, D. Hickey, F. Cleary, M. I. Ali
View a PDF of the paper titled Assisted Spatial Cognition Through Vision-Language Models, by H. Riaz and 5 other authors
View PDF
HTML (experimental)
Abstract:Multimodal AI, powered by Large Language Models (LLMs) and Vision-Language Models (VLMs), is transforming assistive technologies by enabling simultaneous processing of visual and textual data. This advancement holds significant promise for over 43 million visually impaired and neuro-divergent individuals worldwide who face persistent challenges in navigating indoor and outdoor environments due to limited spatial awareness and insufficient environmental cues. Existing navigation aids often lack comprehensive 3D scene understanding, relying on constrained route-based strategies that hinder user autonomy. In this paper, we introduce a novel end-to-end framework that integrates LLMs, VLMs and digital twin technologies to deliver a spatially cognitive navigation support for visually impaired and neuro-divergent users. Our system captures video input via standard mobile phone cameras, and employs SLAM3R to generate dense 3D point clouds from monocular RGB sequences in real-time. Our custom post-processing algorithm ensures accurate point cloud alignment across multiple viewpoints without requiring predefined reference points. This enhances the capabilities of SpatialLM to produce structured 3D representations, including architectural elements and oriented object bounding boxes. The enriched spatial data is then processed by a locally deployed LLM, which interprets 3D contexts to generate detailed scene descriptions and precise distance measurements between users and surrounding objects. We evaluated our approach across diverse video scenarios featuring various perspectives, looped walking views and captured in multiple environments. The evaluation results demonstrate consistent accuracy in 3D scene interpretation and object localisation, underscoring the potential of our system as a transformative assistive navigation solution that combines advanced visual perception with spatial reasoning
Subjects:
Artificial Intelligence (cs.AI)
Cite as:
arXiv:2609.12747 [cs.AI]
(or
arXiv:2609.12747v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.12747
Focus to learn more
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Jaime Boanerjes Fernandez Roblero Dr [view email]
[v1]
Fri, 11 Sep 2026 11:58:59 UTC (6,456 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Assisted Spatial Cognition Through Vision-Language Models, by H. Riaz and 5 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.AI
new
|
recent
| 2026-09
Change to browse by:
cs
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
export BibTeX citation
Loading...
BibTeX formatted citation
×
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? |
Disable MathJax (What is MathJax?)