AdaVid: Adaptive Video-Language Pretraining.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Contrastive video-language pretraining yields robust video representations but faces deployment challenges on resource-constrained edge devices due to high computational costs.
• Current models are limited to processing short video clips (4-64 frames), restricting their applicability to longer video analysis.

Purpose of the Study:

• Introduce AdaVid, a flexible framework for learning efficient, adaptive video encoders.
• Enable dynamic adjustment of computational footprint based on available resources for edge deployment.
• Improve performance and efficiency for both short and long video understanding tasks.

Main Methods:

• Developed an adaptive transformer block inspired by Matryoshka Representation Learning, allowing dynamic adjustment of hidden embedding dimensions at inference.
• Proposed a lightweight hierarchical network to aggregate features from short clips for processing longer videos.
• Trained AdaVid-EgoVLP on the Ego4D dataset for video-narration tasks and evaluated on Diving48 and other long video benchmarks.

Main Results:

• AdaVid-EgoVLP matched standard EgoVLP performance on short video-language tasks using half the compute.
• AdaVid outperformed EgoVLP with equal computational resources on short video benchmarks.
• Demonstrated effective trade-offs between frame count and compute on the Diving48 benchmark, enabling more frames within limits.
• Achieved a strong balance between compute efficiency and accuracy on long video benchmarks using the hierarchical network.

Conclusions:

• AdaVid offers a flexible and efficient solution for deploying advanced video encoders on edge devices.
• The adaptive transformer block and hierarchical network effectively manage computational resources for diverse video lengths and tasks.
• AdaVid significantly advances the efficiency and applicability of video-language pretraining in real-world, resource-limited scenarios.