Leave It to Large Language Models! Correction and Planning with Memory Integration.

Area of Science:

• Artificial Intelligence
• Robotics
• Natural Language Processing

Background:

• Traditional Vision-Language Navigation (VLN) relies on costly manual labeling for task planning in new environments.
• Large Language Models (LLMs) offer commonsense knowledge but struggle with dynamic execution in embodied tasks due to obstacles and state changes.
• Single-plan approaches in VLN are brittle and prone to failure from minor execution errors.

Purpose of the Study:

• To develop a more efficient and cost-effective method for agents to comprehend and execute natural language instructions in embodied tasks.
• To leverage LLMs' reasoning capabilities for dynamic planning and adaptation during task execution.
• To improve the performance and success rate of agents in few-shot learning scenarios for VLN.

Main Methods:

• Proposed a novel approach named Correction and Planning with Memory Integration (CPMI), centered on a Large Language Model (LLM).
• Integrated auxiliary modules within the LLM-centric planner to provide memory and generalized experience mechanisms.
• Facilitated dynamic planning and real-time adjustments during task execution based on environmental feedback.

Main Results:

• CPMI demonstrated superior performance in few-shot learning scenarios on public datasets.
• The approach significantly improved the efficiency of task completion.
• The success rate of agents executing natural language instructions was substantially increased.

Conclusions:

• The CPMI approach effectively enhances LLM capabilities for embodied tasks by enabling dynamic planning and adaptation.
• Memory and generalized experience mechanisms are crucial for improving agent performance during execution.
• CPMI offers a promising direction for more robust and efficient embodied AI agents, particularly in low-data regimes.