Joint Multimodal Embedding and Backtracking Search in Vision-and-Language Navigation

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Multimodal intelligent tasks integrating vision and language are gaining prominence.
• Vision-and-language navigation (VLN) requires aligning and grounding image and text data for real-time task status perception.

Purpose of the Study:

• To propose a novel deep neural network model, JMEBS, for enhanced performance in vision-and-language navigation tasks.
• To improve task success rates and optimize navigation paths through advanced embedding and search algorithms.

Main Methods:

• Developed a transformer-based joint multimodal embedding module for JMEBS, utilizing both multimodal and temporal contexts.
• Implemented backtracking-enabled greedy local search (BGLS) with a novel global scoring method for action selection and trajectory evaluation.
• Evaluated the model using the Matterport3D Simulator and room-to-room (R2R) benchmark datasets.

Main Results:

• The JMEBS model demonstrated improved task success rates and optimized navigation paths compared to existing models.
• The novel global scoring method effectively improved performance by comparing partial trajectories with natural language instructions.
• Experimental results validated the model's effectiveness across various operations on benchmark datasets.

Conclusions:

• The proposed JMEBS model offers a significant advancement in vision-and-language navigation.
• The integration of joint multimodal embedding and backtracking search effectively addresses key challenges in VLN.
• JMEBS provides a robust framework for future research in embodied AI and multimodal understanding.