Cross-modality integration framework with prediction, perception and discrimination for video anomaly detection.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Multimedia Security

Background:

• Video anomaly detection is crucial for public security, aiming to identify deviations from normal patterns.
• Existing methods often overlook textual information, relying primarily on visual data.
• Integrating textual semantics can significantly improve the perception and detection of anomalies.

Purpose of the Study:

• To propose a novel cross-modality integration framework (CIForAD) for video anomaly detection.
• To effectively combine visual and textual modalities for enhanced prediction, perception, and discrimination of anomalies.
• To advance the state-of-the-art in anomaly detection by leveraging multi-modal information.

Main Methods:

• A Feature Fusion Prediction (FUP) module fuses visual and textual features to predict target regions, amplifying discriminative distance.
• An Image-Text Semantic Perception (ISP) module assesses semantic consistency using fine-grained visual and textual features with a local training/global inference strategy.
• A Self-Supervised Time Attention Discrimination (TAD) module explores inter-frame relationships to distinguish abnormal sequences.

Main Results:

• The proposed CIForAD framework demonstrates superior performance in video anomaly detection.
• Experiments on three challenging benchmarks confirm the effectiveness of the cross-modality approach.
• The integration of textual and visual modalities leads to state-of-the-art results.

Conclusions:

• CIForAD effectively integrates textual and visual information for robust video anomaly detection.
• The framework achieves state-of-the-art performance, outperforming existing methods.
• This multi-modal approach offers significant improvements for public security applications.