Dream to Recall: Imagination-Guided Experience Retrieval for Memory-Persistent Vision-and-Language Navigation

Area of Science:

• Artificial Intelligence
• Robotics
• Computer Vision

Background:

• Vision-and-Language Navigation (VLN) involves agents following instructions in environments.
• Memory-persistent VLN aims for continuous improvement via accumulated experience.
• Current methods struggle with effective memory access and integrating behavioral patterns.

Purpose of the Study:

• To introduce Memoir, a novel approach for memory-persistent VLN.
• To enhance memory retrieval by using an imagined future as a query mechanism.
• To improve navigation by integrating both environmental observations and behavioral histories.

Main Methods:

• Developed a language-conditioned world model for experience encoding and retrieval query generation.
• Implemented Hybrid Viewpoint-Level Memory to store observations and behavioral patterns linked to viewpoints.
• Utilized an experience-augmented navigation model with specialized encoders for knowledge integration.

Main Results:

• Achieved significant improvements across diverse memory-persistent VLN benchmarks.
• Demonstrated a 5.4% SPL gain on IR2R compared to the best baseline.
• Reported an 8.3× training speedup and 74% inference memory reduction.

Conclusions:

• Predictive retrieval of environmental and behavioral memories enhances navigation effectiveness.
• Memoir's imagination-guided paradigm shows substantial potential for future advancements.
• The approach offers a more efficient and effective solution for complex navigation tasks.