Two-Stream Retentive Long Short-Term Memory Network for Dense Action Anticipation

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Human action analysis in long-range videos is crucial for applications like surveillance, autonomous driving, and human-computer interaction.
• Current research predominantly focuses on short-range videos, predicting single actions or forecasting immediate future actions.
• A gap exists in predicting a sequence of actions and their temporal extents from incomplete video data.

Purpose of the Study:

• To propose a novel method for forecasting a series of human actions and their durations from partial video observations.
• To address the limitations of existing methods in handling long-range video action prediction.
• To develop a system capable of understanding complex, sequential human behaviors over extended periods.

Main Methods:

• Feature extraction from both frame sequences and associated label sequences.
• Introduction of a retentive memory module to capture salient temporal and channel-wise information.
• Utilizing deep learning architectures for sequence modeling and prediction.

Main Results:

• The proposed method demonstrates comparable performance against state-of-the-art techniques on benchmark datasets (Breakfast and 50 Salads).
• Successful forecasting of action series and their durations was achieved, indicating robustness in long-range video understanding.
• The retentive memory module proved effective in enriching feature representations at critical time steps and feature channels.

Conclusions:

• The developed method offers a significant advancement in long-range human action forecasting.
• This approach provides a more comprehensive understanding of sequential actions compared to existing short-range focused methods.
• The findings have implications for improving surveillance systems, autonomous driving perception, and human-computer interaction interfaces.