Efficient Transformer-Based Compressed Video Modeling via Informative Patch Selection.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Transformer-based models excel in video recognition but suffer from high computational costs.
• Existing video compression techniques effectively reduce redundancy by prioritizing informative content.

Purpose of the Study:

• To develop an efficient method for reducing the inference cost of Transformer-based video recognition models.
• To improve the accuracy-inference cost trade-off for practical video analysis applications.

Main Methods:

• Propose Informative Patch Selection (IPS) to exclude redundant patches from Transformer inputs.
• Calculate patch redundancy using motion and residual information from compressed video decoding.
• Implement a dynamic reduction of inference cost based on input data without policy models or extra loss terms.

Main Results:

• Extensive experiments on action recognition benchmarks show significant improvements in the accuracy-inference cost trade-off.
• The proposed method achieves performance comparable to existing methods despite its simplicity.
• IPS effectively reduces computational load without compromising model accuracy.

Conclusions:

• Informative Patch Selection (IPS) offers an efficient solution to the high inference cost of Transformer-based video models.
• The method's dynamic, data-driven approach provides flexibility and broad applicability.
• IPS enhances the practical usability of advanced video recognition models.