Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Egocentric action anticipation is crucial for understanding human behavior from first-person perspectives.
• Predicting future actions and object interactions from egocentric videos remains a significant challenge.

Purpose of the Study:

• To develop a novel deep learning architecture for accurate egocentric action anticipation.
• To improve the prediction of future actions and object interactions in egocentric videos.

Main Methods:

• Proposed the Rolling-Unrolling LSTM architecture with two LSTMs for past summarization and future inference.
• Introduced a Sequence Completion Pre-Training technique to enhance LSTM focus on sub-tasks.
• Implemented a Modality Attention (MATT) mechanism for fusing multi-modal predictions (RGB, optical flow, object features).

Main Results:

• Achieved state-of-the-art performance on the EPIC-Kitchens egocentric action anticipation challenge (2019).
• Demonstrated competitive results on the ActivityNet dataset compared to unsupervised pre-training methods.
• Showcased generalization capabilities to early action recognition and action recognition tasks.

Conclusions:

• The Rolling-Unrolling LSTM architecture significantly advances egocentric action anticipation.
• The proposed method offers robust performance across multiple datasets and related tasks.
• Open-sourced code, models, and features to foster further research in this domain.