Unseen From Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language...

Area of Science:

• Artificial Intelligence
• Robotics
• Computer Vision

Background:

• Data scarcity is a major limitation in vision-language navigation (VLN), hindering agent generalization to new environments.
• Existing methods use simulator or web data, which have limited diversity or require significant manual cleaning.
• This limits the ability of agents to navigate effectively in real-world, unseen scenarios.

Purpose of the Study:

• To introduce a novel Rewriting-driven AugMentation (RAM) paradigm for VLN.
• To overcome data scarcity by generating diverse, unseen observation-instruction pairs from existing data.
• To improve the generalization capabilities of VLN agents in a simulator-free and labor-saving manner.

Main Methods:

• Object-enriched observation rewriting using vision-language models (VLMs) and large language models (LLMs) to create diverse scene descriptions.
• Text-to-image generation models (T2IMs) synthesize new observations based on rewritten descriptions.
• Observation-contrast instruction rewriting uses LLMs to align new instructions with synthesized observations.
• A mixing-then-focusing training strategy with random observation cropping enhances data diversity and reduces noise.

Main Results:

• The RAM paradigm successfully generates novel observation-instruction pairs for VLN training.
• Experiments demonstrate superior performance and enhanced generalization on discrete (R2R, REVERIE, R4R) and continuous (R2R-CE) VLN datasets.
• The method effectively improves agent performance in unseen environments.

Conclusions:

• Rewriting-driven AugMentation (RAM) is an effective approach to address data scarcity in VLN.
• The proposed method offers a simulator-free and labor-saving solution for data augmentation.
• RAM significantly improves the generalization ability of VLN agents, paving the way for more robust navigation systems.