Egocentric Perception of Walking Environments Using an Interactive Vision-Language System.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Robotics

Background:

• Large language models (LLMs) offer contextual scene understanding beyond computer vision.
• Embodied intelligence and robotics benefit from enhanced perception systems.

Purpose of the Study:

• Develop a multimodal vision-language system for egocentric visual perception.
• Enable personalized image captioning and audio feedback for real-world navigation.

Main Methods:

• Trained transformer-based vision-language models using causal language modeling.
• Utilized a custom dataset of 43,055 image-text pairs for few-shot image captioning.
• Developed a speech synthesis model and user interface for audio feedback and personalized captions via user prompts.

Main Results:

• Generated detailed image captions (avg. 10 words) with high ROUGE-L score (43.9%) and low word error rate (28.1%).
• Achieved end-to-end processing time of 2.2 seconds.
• Demonstrated effective personalization of captions through user prompts.

Conclusions:

• The multimodal system provides accurate, detailed scene descriptions.
• Personalized captions optimize human-AI interaction for environmental understanding and navigation.
• This work advances embodied AI by integrating human cognition into generative models.