A Short Video Classification Framework Based on Cross-Modal Fusion.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Online short videos present significant challenges for content classification and management.
• Traditional video classification methods relying solely on visual features are computationally intensive and may lack accuracy.
• Existing single-modality approaches struggle to meet specific scenario accuracy requirements.

Purpose of the Study:

• To develop an efficient short video categorization architecture for visual sensor systems.
• To improve the accuracy of short video classification by integrating multiple data modalities.
• To reduce the computational load associated with frame-by-frame video processing.

Main Methods:

• Utilized a self-attention mechanism to extend image frames into a 3D space-time representation.
• Extracted video features using the Timesformer network by mapping image patches to an embedding layer.
• Extracted text features from subtitles using the Bidirectional Encoder Representation from Transformers (BERT) model.
• Implemented a cross-modal fusion strategy to combine video and text features for classification.

Main Results:

• The proposed cross-modal fusion framework significantly outperformed baseline video classification methods.
• The architecture effectively classifies short videos by jointly analyzing visual and textual information.
• Achieved improved accuracy in short video classification tasks compared to single-modality approaches.

Conclusions:

• The developed framework offers a superior approach to short video classification in sensor systems.
• Cross-modal fusion of visual and text features is a promising strategy for enhancing video analysis.
• The method provides an efficient and accurate solution for managing and classifying large volumes of short video content.