Interactive 3D scene graph alignment for zero-shot object navigation

Area of Science:

• Robotics
• Computer Vision
• Artificial Intelligence

Background:

• Embodied agents face challenges navigating unknown, dynamic 3D environments using only visual input.
• Current methods struggle with robust reasoning and interaction in complex, unseen spaces.

Purpose of the Study:

• To develop a novel perception framework, Interactive 3D Scene Graph (I3DSG), for enhanced visual navigation in complex 3D environments.
• To enable embodied agents to reason about missing information and feasible interactions in unseen environments.

Main Methods:

• Constructing a Bird's Eye View (BEV)-based scene graph enriched with geometric layout, semantic relationships, and actionable affordances.
• Integrating open-vocabulary object detection and large language models to align commonsense knowledge with real-time observations.
• Optimizing action selection using graph-based grid decision scoring for consistent path planning.

Main Results:

• Achieved superior performance in success rate and SPL compared to existing methods on standard benchmarks.
• Demonstrated effective generalization to physical environments using a real-world robot (Scout ROS2).
• Validated robustness against sensor noise and illumination variations in real-world deployments.

Conclusions:

• I3DSG provides a robust framework for embodied agents to navigate complex 3D environments.
• The integration of commonsense knowledge and real-time visual data significantly improves navigation capabilities.
• The approach is effective in both simulated and real-world scenarios, paving the way for more capable autonomous systems.