Embodied Spatial Affordance: Spatial-Aware Affordance Learning for Embodied Navigation and Manipulation

Area of Science:

• Robotics and Artificial Intelligence
• Computer Vision
• Human-Robot Interaction

Background:

• Embodied agents require understanding spatial context and object affordances for navigation and manipulation.
• Current Vision-Language Models (VLMs) struggle with precise spatial understanding and affordance localization from images, limiting their application in robotics.
• Bridging the gap between high-level reasoning and low-level actionable commands is crucial for embodied AI.

Purpose of the Study:

• To introduce EspA, a novel image-to-keypoint model for spatial-aware affordance learning.
• To enable precise pixel-level localization of both object and free space affordances directly from 2D image inputs.
• To improve the translation of language instructions into actionable 3D coordinates for embodied agents.

Main Methods:

• Developed a hierarchical vision-language architecture for joint reasoning of object and free space affordances.
• Introduced the Embodied Spatial Affordance (ESA) dataset with fine-grained annotations for embodied interactions.
• Implemented an image-to-keypoint approach to predict affordance keypoints and project them into 3D space using depth information.

Main Results:

• EspA demonstrates superior performance in predicting object and free space affordances compared to state-of-the-art VLMs.
• The model shows enhanced capabilities in real-world embodied navigation and manipulation tasks.
• Achieved significant improvements in image-based spatial reasoning for embodied agents.

Conclusions:

• EspA provides a scalable solution for translating high-level instructions into low-level actionable affordances for embodied agents.
• The proposed method advances embodied AI by enabling more robust and versatile interaction with physical environments.
• The publicly available dataset and code will foster future research in embodied spatial understanding.