Action Anticipation Using Pairwise Human-Object Interactions and Transformers.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Anticipating human actions involving objects is crucial for applications like automated driving, robotics, and smart homes.
• Current methods often require object affordance labels, limiting their real-world applicability.
• Representing human-object interactions solely through visual features offers a more flexible approach.

Purpose of the Study:

• To develop a novel method for anticipating human actions by representing human-object interactions using only visual features.
• To improve the accuracy and robustness of action anticipation models, particularly in complex, unscripted environments.
• To introduce a multi-modal transformer (MM-Transformer) model for effective spatio-temporal action anticipation.

Main Methods:

• Representing pairwise human-object (HO) interactions using visual features.
• Employing cross-correlation to capture second-order statistics of HO pairs for holistic frame representation.
• Utilizing a transformer model for temporal aggregation of frame-wise HO representations.

Main Results:

• Cross-correlation based frame representation outperforms attention-based and other second-order methods for action anticipation.
• Transformer models show superior performance in temporal aggregation of HO representations compared to other temporal networks.
• The proposed MM-Transformer model demonstrates effectiveness on procedural and unscripted datasets, including long-term anticipation.

Conclusions:

• Combining human-object visual representation with MM-Transformers provides a powerful framework for action anticipation.
• The approach eliminates the need for explicit object affordance labels, making it more practical for real-world scenarios.
• This research advances the capability of AI systems to understand and predict human behavior in dynamic environments.