PanoGen++: Domain-adapted text-guided panoramic environment generation for vision-and-language navigation

Area of Science:

• Artificial Intelligence
• Robotics
• Computer Vision

Background:

• Vision-and-language navigation (VLN) agents require extensive training data to navigate 3D environments using natural language.
• Data scarcity is a significant bottleneck hindering progress and generalization in VLN research.

Purpose of the Study:

• To introduce PanoGen++, a novel framework for generating diverse and relevant panoramic training environments for VLN tasks.
• To address the challenge of limited training data in VLN by creating synthetic yet pertinent environments.

Main Methods:

• Utilized pre-trained diffusion models with domain-specific fine-tuning and parameter-efficient low-rank adaptation.
• Investigated masked image inpainting for novel environment creation and recursive image outpainting for spatial relationship learning.

Main Results:

• Achieved a 2.44% success rate increase on the Room-to-Room (R2R) leaderboard.
• Improved performance by 0.63% on the Room-for-Room (R4R) validation unseen set.
• Enhanced goal progress by 0.75 meters on the Cooperative Vision-and-Dialog Navigation (CVDN) validation unseen set.

Conclusions:

• PanoGen++ effectively augments training data diversity and relevance for VLN.
• The framework leads to improved generalization and efficacy of navigation agents.
• Demonstrated significant performance gains across multiple benchmark VLN datasets.