A retrieval-augmented framework enabling VLM spatial awareness for object-centric robot manipulation

Area of Science:

• Robotics
• Artificial Intelligence
• Computer Vision

Background:

• Vision-language models (VLMs) struggle with precise spatial reasoning for robotic manipulation.
• Existing VLMs lack the intrinsic spatial intelligence for object placement and orientation tasks.

Purpose of the Study:

• Introduce Retrieval-Augmented Manipulation (RAM) to bridge the semantic-to-geometric gap in robotic manipulation.
• Equip general-purpose vision foundation models with spatial reasoning capabilities for complex tasks.

Main Methods:

• Developed an object-centric framework (RAM) grounding abstract concepts into 3D representations.
• Augmented VLMs with grounded 3D information to decompose instructions into precise subgoals.
• Utilized a real-world robot for zero-shot execution of manipulation tasks.

Main Results:

• RAM successfully executed complex spatial language instructions in a zero-shot setting.
• Demonstrated spatially aware manipulation from a single 2D image and adaptive replanning.
• Validated generalization to unseen objects and robustness to shape variations and occlusions on the CO3D dataset.

Conclusions:

• RAM provides a structured bridge between semantic intent and geometric execution for robotic systems.
• This framework is a critical step toward developing more physically intelligent and general-purpose robots.
• The object-centric approach enhances VLM spatial reasoning for real-world manipulation challenges.