A Modular Vision Language Navigation and Manipulation Framework for Long Horizon Compositional Tasks in Indoor...

Area of Science:

• Robotics
• Artificial Intelligence
• Computer Vision

Background:

• Existing end-to-end frameworks struggle with long-horizon, compositional tasks in household settings.
• There's a need for robust methods that handle diverse objects, realistic instructions, and non-reversible state changes.

Purpose of the Study:

• Propose MoViLan (Modular Vision and Language), a novel framework for executing visually grounded natural language instructions for indoor household tasks.
• Address limitations in current approaches for complex manipulation and navigation tasks.

Main Methods:

• Developed a modular approach separating vision and language processing, reducing reliance on strictly aligned training data.
• Introduced a novel geometry-aware mapping technique for cluttered indoor environments.
• Created a generalized language understanding model for household instruction following.

Main Results:

• MoViLan significantly increases success rates for long-horizon, compositional tasks.
• Demonstrated superior performance over recent works on the ALFRED benchmark dataset.
• The modular design allows for more tractable training with separate vision and language datasets.

Conclusions:

• The MoViLan framework offers a more effective and flexible solution for robot instruction following in complex home environments.
• Modular design represents a significant departure from traditional end-to-end methods, enabling better generalization and training efficiency.