Building an Open-Vocabulary Video CLIP Model With Better Architectures, Optimization and Data.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Contrastive Language-Image Pretraining (CLIP) excels in zero-shot image recognition but has limited application in video recognition.
• Adapting CLIP for video requires capturing spatial-temporal dynamics while maintaining generalization capabilities.

Purpose of the Study:

• To develop Open-VCLIP++, a framework that enhances CLIP for robust zero-shot video recognition.
• To address the challenge of training with zero historical data using novel optimization techniques.
• To improve the transfer of CLIP's capabilities to the video domain through fine-grained video descriptions.

Main Methods:

• Open-VCLIP++ minimally modifies CLIP to incorporate spatial-temporal video features.
• Interpolated Weight Optimization is introduced for effective training and testing without historical data.
• Large language models generate detailed video descriptions, aligned with video features for improved CLIP transfer.

Main Results:

• Open-VCLIP++ achieves state-of-the-art zero-shot accuracy on action recognition datasets: 88.1% (UCF), 58.7% (HMDB), and 81.2% (Kinetics-600).
• The method significantly outperforms existing approaches by up to 12.3%.
• Competitive performance is demonstrated on the MSR-VTT dataset for video-text retrieval with reduced fine-tuning data.

Conclusions:

• Open-VCLIP++ presents a highly effective and generalizable framework for zero-shot video recognition.
• The proposed Interpolated Weight Optimization and description alignment techniques are crucial for success.
• This work opens new avenues for leveraging large pre-trained models in video understanding tasks.