Uni-AdaFocus: Spatial-Temporal Dynamic Computation for Video Recognition.

Area of Science:

• Computer Vision and Machine Learning
• Artificial Intelligence
• Data Science

Background:

• Video understanding research faces challenges with computational inefficiency due to data redundancy.
• Spatial redundancy, where informative regions shift smoothly across frames, is a key area for optimization.
• Existing methods often process entire video frames, leading to unnecessary computational load.

Purpose of the Study:

• To develop a computationally efficient video understanding approach by addressing data redundancy.
• To introduce AdaFocus, a spatially adaptive method that focuses on task-relevant image patches.
• To extend AdaFocus into Uni-AdaFocus, integrating spatial, temporal, and sample-wise dynamic computation for comprehensive efficiency.

Main Methods:

• AdaFocus employs a lightweight encoder and a policy network to identify and process informative spatial patches.
• Selected patches are then analyzed by a high-capacity deep network for final predictions, enabling end-to-end training.
• Uni-AdaFocus further incorporates temporal and sample-wise redundancy reduction, allocating computation dynamically across frames and videos.

Main Results:

• AdaFocus demonstrates efficient parallel processing on GPUs and effective end-to-end training.
• Uni-AdaFocus achieves significant computational efficiency improvements across seven benchmark datasets and three real-world scenarios.
• The Uni-AdaFocus framework is compatible with off-the-shelf backbone models like TSM and X3D.

Conclusions:

• AdaFocus and Uni-AdaFocus offer a significant advancement in efficient video understanding.
• These methods provide a flexible and general framework for reducing computational costs in video recognition.
• The proposed techniques show superior performance compared to existing baselines across diverse applications.