EVA02-AT: Egocentric Video-Language Understanding with Spatial-Temporal Rotary Positional Embeddings and Symmetric...

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Egocentric video-language understanding requires efficient and accurate spatial-temporal modeling.
• Current methods struggle with high pre-training costs, ineffective spatial-temporal encoding, and imprecise learning objectives.
• Existing approaches often use multi-stage pre-training and manually split positional embeddings, limiting feature interaction.

Purpose of the Study:

• Introduce EVA02-AT, a suite of EVA02-based foundation models for egocentric video understanding.
• Address limitations in pre-training efficiency, spatial-temporal encoding, and retrieval learning objectives.
• Achieve state-of-the-art performance on egocentric video-language tasks with improved efficiency.

Main Methods:

• Efficiently transfer an image-based CLIP model to a unified video encoder via single-stage pretraining.
• Introduce spatial-temporal rotary positional embeddings and joint attention for effective encoding of spatial and temporal information.
• Propose Symmetric Multi-Similarity (SMS) loss and a novel training framework for precise learning objectives in multi-instance retrieval.

Main Results:

• EVA02-AT achieves state-of-the-art performance on Ego4D, EPIC-Kitchens-100, and Charades-Ego datasets.
• Demonstrates superior results in both zero-shot and fine-tuning settings for egocentric video-language tasks.
• Models utilizing SMS loss show significant performance gains on multi-instance retrieval benchmarks.

Conclusions:

• EVA02-AT provides an efficient and effective solution for egocentric video-language understanding.
• The proposed spatial-temporal rotary positional embeddings and SMS loss significantly improve model performance.
• The publicly available code and models facilitate further research in this domain.