Correctable Landmark Discovery via Large Models for Vision-Language Navigation.

Area of Science:

• Artificial Intelligence
• Robotics
• Computer Vision

Background:

• Vision-Language Navigation (VLN) agents struggle with accurate modality alignment, especially in novel environments due to limited training data.
• Existing VLN methods lack sufficient open-world knowledge for aligning linguistic landmarks with visual observations.

Purpose of the Study:

• To propose a novel VLN paradigm, COrrectable LaNdmark DiScOvery via Large ModEls (CONSOLE), for improved landmark discovery and navigation.
• To enhance VLN agents' ability to align language instructions with visual observations in unexplored scenes.

Main Methods:

• CONSOLE frames VLN as an open-world sequential landmark discovery problem.
• It leverages ChatGPT for commonsense co-occurrence knowledge and CLIP for landmark discovery, guided by these priors.
• A learnable cooccurrence scoring module corrects prior noise using actual observations, and an observation enhancement strategy integrates corrected landmark features for action decisions.

Main Results:

• CONSOLE demonstrates significant superiority over strong baselines across multiple VLN benchmarks (R2R, REVERIE, R4R, RxR).
• The framework establishes new state-of-the-art results on the R2R and R4R benchmarks, particularly in unseen scenarios.
• CONSOLE effectively improves landmark discovery and modality alignment in challenging, unexplored environments.

Conclusions:

• CONSOLE offers a powerful new paradigm for Vision-Language Navigation by integrating large language models for enhanced landmark discovery.
• The proposed correctable landmark discovery scheme and observation enhancement strategy significantly boost VLN performance, especially in complex and unseen environments.
• This work paves the way for more robust and adaptable VLN agents capable of navigating diverse and open-world settings.