Integrating Affordances and Attention Models for Short-Term Object Interaction Anticipation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Short-Term object-interaction Anticipation (STA) is crucial for AI systems like wearable assistants and robots to understand user intent and provide timely support.
• Current STA methods require improvement to accurately predict object interactions from egocentric video data.

Purpose of the Study:

• To enhance the performance of short-term object-interaction anticipation (STA) predictions.
• To introduce novel attention-based architectures and modules for grounding STA predictions in human behavior and environmental context.

Main Methods:

• Proposed STAformer and STAformer++: attention-based architectures with frame-guided temporal pooling, dual image-video attention, and multiscale feature fusion.
• Introduced environment affordance modeling for persistent interaction memory within scenes.
• Developed adaptive fusion methods for integrating affordances with end-to-end predictions.
• Integrated interaction hotspot prediction from hand and object trajectories to refine STA localization.

Main Results:

• Significant improvements in Overall Top-5 mAP, with gains up to 23% on Ego4D and 31% on curated EPIC-Kitchens STA labels.
• Demonstrated the effectiveness of the proposed attention architectures and affordance-grounded modules.

Conclusions:

• The novel STAformer architectures and affordance-based modules substantially advance short-term object-interaction anticipation.
• The released code, annotations, and affordances will facilitate future research in egocentric video understanding and human-AI interaction.