Self-Supervised Learning from Untrimmed Videos via Hierarchical Consistency

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Self-supervised learning for spatio-temporal representations often uses manually trimmed videos, limiting visual diversity and performance.
• Untrimmed videos offer richer content but pose challenges for representation learning.

Purpose of the Study:

• To improve video representations by utilizing natural, untrimmed videos.
• To develop a framework that learns a hierarchy of temporal consistencies.

Main Methods:

• Propose Hierarchical Consistency (HiCo++) learning framework.
• Learn visual consistency (short-span, visually similar clips) via contrastive learning.
• Learn topical consistency (long-span, topic-related clips) via a topical classifier.
• Employ a gradual sampling algorithm for hierarchical consistency learning.

Main Results:

• HiCo++ generates stronger representations from untrimmed videos.
• The framework improves representation quality when applied to trimmed videos.
• Demonstrates superiority over standard contrastive learning on untrimmed videos.

Conclusions:

• HiCo++ effectively leverages untrimmed videos for enhanced self-supervised representation learning.
• The hierarchical consistency approach offers a significant advancement over existing methods.
• This framework opens new possibilities for learning from diverse, uncurated video data.