STAC: Spatial-Temporal Attention on Compensation Information for Activity Recognition in FPV

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Egocentric activity recognition in first-person video (FPV) is challenging due to background noise and small, fine-grained objects.
• Traditional third-person action recognition methods are insufficient for FPV challenges.

Purpose of the Study:

• To develop a novel deep learning architecture for robust egocentric activity recognition.
• To address the limitations of background noise and object scale variations in FPV.

Main Methods:

• Developed a two-stream Convolutional Neural Network-Recurrent Attention Network (CNN-RAN) architecture named STAC.
• Implemented size compensation for data augmentation and optical flow compensation to reduce motion noise.
• Utilized spatial-temporal attention mechanisms for object-centric and motion-focused feature extraction.

Main Results:

• The STAC model demonstrated superior performance by effectively handling noise and object scale.
• Achieved state-of-the-art results on two benchmark egocentric datasets.
• Ablation studies confirmed the complementarity and effectiveness of the STAC model's components.

Conclusions:

• The proposed STAC model offers a significant advancement in egocentric activity recognition.
• The novel architecture and attention mechanisms provide a robust solution for FPV analysis.
• This work paves the way for more accurate and reliable human activity understanding from first-person perspectives.