Learning Visual Affordance Grounding From Demonstration Videos.

Area of Science:

• Computer Vision
• Robotics
• Human-Computer Interaction

Background:

• Visual affordance grounding identifies interaction regions between people and objects for applications like robot grasping.
• Current methods struggle with multiple interaction possibilities on objects and multiple human actions within the same region.

Purpose of the Study:

• To propose a novel Hand-Aided Affordance Grounding Network (HAG-Net) to address limitations in current visual affordance grounding methods.
• To leverage hand position and action cues from demonstration videos to refine interaction region localization.

Main Methods:

• HAG-Net employs a dual-branch architecture processing video and object data separately.
• The video branch uses hand-aided attention and Long Short-Term Memory (LSTM) for action feature aggregation.
• The object branch incorporates a Semantic Enhancement Module (SEM) and distillation loss for knowledge transfer.

Main Results:

• Evaluations on two datasets demonstrate HAG-Net achieves state-of-the-art performance in visual affordance grounding.
• The method effectively disambiguates multiple interaction possibilities on objects and refines localization using hand cues.

Conclusions:

• HAG-Net successfully improves visual affordance grounding by integrating hand-centric information from demonstration videos.
• The proposed network offers a more robust solution for segmenting human-object interaction regions in complex scenarios.