Generalizable and Slim Affordance Reasoning for Task-oriented Manipulation

Area of Science:

• Artificial Intelligence
• Robotics
• Computer Vision

Background:

• Object affordance reasoning is crucial for AI task planning but current models lack generalizability.
• Large Language Models (LLMs) offer reasoning capabilities but are difficult to deploy locally.
• Existing methods struggle with novel scenarios and efficient, on-device deployment.

Purpose of the Study:

• To enhance the generalizability of affordance reasoning from perception.
• To develop an efficient and deployable model for task-oriented manipulations.
• To create a large-scale dataset for training and evaluating affordance reasoning models.

Main Methods:

• Introduced LVIS-Aff, a large-scale dataset with 1,496 tasks and 119k images.
• Developed Afford-X, an end-to-end trainable model using Verb Attention and Bi-Fusion modules.
• Evaluated model performance against non-LLM methods and previous work.

Main Results:

• Afford-X achieved up to a 12.1% performance improvement over non-LLM methods.
• Demonstrated a 1.2% enhancement compared to the previous conference paper.
• The model has a compact 187M parameter size and is ~50x faster than GPT-4V API inference.

Conclusions:

• The research shows potential for efficient, generalizable affordance reasoning models deployable on local devices.
• Afford-X enables effective task-oriented object grasping for robots in diverse environments.
• The findings support the development of AI systems for real-world manipulation tasks.