History-Guided Prompt Generation for Vision-and-Language Navigation

Area of Science:

• Embodied artificial intelligence
• Computer vision
• Natural language processing

Background:

• Vision-and-language navigation (VLN) relies on historical observations for contextual knowledge.
• Current VLN methods fail to explicitly connect historical context with the current environment.
• Adaptive learning of environment-specific clues is often overlooked in existing approaches.

Purpose of the Study:

• To enhance agent perception in vision-and-language navigation by adaptively mining relevant historical information.
• To propose a novel history-guided prompt generation (HGPG) framework for VLN.
• To improve generalization to unknown environments by sharing learned representations across tasks.

Main Methods:

• Developed an entropy-based history acquisition module to assess the necessity of historical information.
• Implemented a prompt generation module that converts historical context into compact prompt vectors using a learned token library.
• Employed a shared token library across diverse navigation tasks to capture common features and enhance generalization.

Main Results:

• The HGPG framework demonstrated significant effectiveness on four mainstream VLN benchmarks (R2R, REVERIE, SOON, R2R-CE).
• The proposed method successfully enhances the agent's perception of the current environment by leveraging historical data.
• Sharing the token library improved generalization capabilities to previously unseen environments.

Conclusions:

• The history-guided prompt generation framework offers a promising approach for improving vision-and-language navigation.
• Adaptive mining of historical information is crucial for robust and generalized navigation.
• The HGPG method provides a more efficient and effective way for agents to utilize past experiences in dynamic environments.