Multi-Moments in Time: Learning and Interpreting Models for Multi-Action Video Understanding.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Videos often contain multiple actions occurring simultaneously or sequentially within seconds.
• Current large-scale video datasets predominantly use single labels, limiting model training for complex events.
• This single-label approach can lead to models being penalized for unlabelled, yet present, actions.

Purpose of the Study:

• Introduce the Multi-Moments in Time dataset (M-MiT) for multi-action recognition.
• Address the challenges of training and analyzing models for multi-action detection.
• Provide baseline results and demonstrate the utility of M-MiT for model development and transfer learning.

Main Methods:

• Developed the Multi-Moments in Time (M-MiT) dataset with over two million action labels for one million three-second videos.
• Adapted loss functions for long-tail multi-label learning to train action recognition models.
• Implemented improved visualization and interpretation methods for multi-label action detection models.

Main Results:

• Established baseline performance for multi-action recognition on the M-MiT dataset.
• Demonstrated effective training strategies for multi-label action detection.
• Showcased the effectiveness of M-MiT for transfer learning to smaller datasets.

Conclusions:

• The M-MiT dataset offers a significant resource for advancing multi-action recognition research.
• Multi-label learning approaches are crucial for accurately interpreting complex video content.
• Models trained on M-MiT show strong generalization capabilities, benefiting downstream tasks.