Temporal Perceiver: A General Architecture for Arbitrary Boundary Detection.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Generic Boundary Detection (GBD) is crucial for long-form video understanding, segmenting videos into coherent units.
• Existing GBD methods often use specialized deep networks for different boundary types (shot, event, scene), lacking a unified approach.
• Previous models face challenges with computational complexity, particularly the quadratic relationship between attention operations and input frames.

Purpose of the Study:

• To introduce Temporal Perceiver, a unified Transformer-based architecture for detecting arbitrary generic boundaries in videos.
• To reduce the computational complexity of attention mechanisms for video processing.
• To enhance the generalization ability of GBD models across diverse datasets.

Main Methods:

• Developed Temporal Perceiver, a Transformer architecture utilizing a small set of latent feature queries as anchors.
• Employed cross-attention blocks to compress video input into a fixed dimension, achieving linear complexity with respect to input frames.
• Introduced boundary and context queries to capture temporal structure, along with an alignment loss to guide query learning and a sparse detection head for direct output.

Main Results:

• Achieved state-of-the-art performance on multiple GBD benchmarks (SoccerNet-v2, Kinetics-GEBD, TAPOS, MovieScenes, MovieNet) using single-stream RGB features.
• Demonstrated significant improvements in average-mAP and average-f1 scores across all tested datasets.
• A class-agnostic Temporal Perceiver model showed comparable accuracy and superior generalization capabilities compared to dataset-specific models.

Conclusions:

• Temporal Perceiver provides a generalized and efficient solution for various types of Generic Boundary Detection.
• The proposed latent query mechanism effectively reduces computational complexity while maintaining high detection accuracy.
• The model's strong performance and generalization highlight its potential as a foundational tool for advanced video understanding tasks.