Interactive image-to-video transfer learning.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Transfer learning from image to video is crucial for action recognition.
• Current methods involve extensive fine-tuning, leading to high computational costs.
• Efficient methods often neglect effective video reasoning with frozen image backbones.

Purpose of the Study:

• To propose an efficient image-to-video transfer learning framework (SDST) for action recognition.
• To enhance the interaction between static and dynamic cues, and spatial and temporal domains.
• To bridge the gap between static visual representations and video-based action recognition tasks.

Main Methods:

• Introducing the Motion Booster Module to extract and fuse motion descriptors with static representations via cross-attention.
• Proposing a lightweight Channel-Aware Multi-Scale Temporal Modelling module for temporal reasoning.
• Enriching a frozen image backbone with these novel components.

Main Results:

• SDST surpasses State-of-the-Art efficient transfer learning methods on benchmarks like Something-Something V1&V2, Diving-48, and Kinetics-400.
• The framework outperforms several fully fine-tuned approaches without additional complexity.
• Demonstrated transferability to vision-language models like CLIP, yielding further performance gains.

Conclusions:

• The proposed SDST framework offers a general and scalable solution for efficient video understanding.
• It effectively addresses the limitations of existing efficient transfer learning methods.
• Highlights the potential for improved action recognition in the era of large vision-language models.