Multimodal Navigation System for Visually Impaired Users Using Environmental Perception and Vision-Language Models

Area of Science:

• Computer Science
• Artificial Intelligence
• Robotics

Background:

• Visually impaired individuals encounter significant obstacles in indoor navigation due to diminished spatial awareness and the absence of real-time semantic cues.
• Existing assistive technologies often require specialized infrastructure, limiting their widespread applicability and effectiveness.

Purpose of the Study:

• To develop and evaluate a multimodal navigation system that leverages vision-language models (VLMs) for enhanced indoor navigation for visually impaired users.
• To provide context-aware, explainable, and infrastructure-independent guidance.

Main Methods:

• The system integrates Real-Time Appearance-Based Mapping (RTAB-Map) for localization, YOLO-World for open-vocabulary object detection, and a lightweight language model for semantic reasoning.
• Evaluation involved the RePOPE benchmark to assess hallucination in vision-language understanding and real-world indoor navigation experiments.

Main Results:

• The proposed system demonstrated improved precision by up to 2.29% and consistently enhanced F1-score compared to baseline VLM approaches.
• Real-world tests confirmed reliable navigation, including multi-floor path planning and obstacle-aware guidance.

Conclusions:

• Integrating environmental perception with language-based reasoning significantly enhances spatial understanding and reduces hallucination in assistive navigation systems.
• The developed multimodal system offers a practical, scalable, and effective solution for improving indoor navigation for the visually impaired.