Markov Progressive Framework, a Universal Paradigm for Modeling Long Videos.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Video models face computational complexity challenges, limiting end-to-end training for long-term temporal semantics.
• Current methods of splitting videos into clips result in fragmented information and hinder long-term semantic understanding.

Purpose of the Study:

• Introduce the Markov Progressive (MaPro) framework for efficient end-to-end video model training.
• Enable modeling of long videos with limited computational resources while preserving temporal information.

Main Methods:

• Design a theoretical framework (MaPro) with a paradigm model using Markov operators.
• Implement MaPro on CNN- and Transformer-based models, enabling multi-step sequential training.
• Ensure multi-step progressive modeling is equivalent to conventional end-to-end modeling.

Main Results:

• MaPro significantly improves performance across different backbones and datasets.
• The SlowOnly network improved by 4.1 mAP on Charades and 2.5 top-1 accuracy on Kinetics.
• TimeSformer performance on Kinetics improved by 2.0 top-1 accuracy with MaPro.

Conclusions:

• MaPro is a general and robust training method for video models.
• Achieves significant performance gains with minimal parameter and computation overhead.
• Effectively addresses the challenge of modeling long-term temporal semantics in videos.