TrCLIP-VAD : Weak supervised video anomaly detection by improving CLIP training with text rewriting

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Existing contrastive language-image pre-training (CLIP) based video anomaly detection (VAD) methods often neglect high-level semantic information like textual descriptions of abnormal events.
• Current VAD techniques commonly augment video frames but fail to vary text inputs, limiting semantic diversity during training.

Purpose of the Study:

• To propose TrCLIP-VAD, a novel CLIP-based VAD model that leverages text rewriting to improve the detection of anomalies.
• To enhance the semantic representation of videos by rewriting existing captions using large language models (LLMs).

Main Methods:

• Developed TrCLIP-VAD, a model that generates source captions for videos and rewrites them using In-Context Learning (ICL) from LLMs.
• Integrated a Local-Global Multi-scale Mamba (LGM-Mamba) module to capture temporal dependencies and fuse visual and textual features.
• Fused rewritten captions with visual features for anomaly detection.

Main Results:

• TrCLIP-VAD achieved state-of-the-art performance on the XD-Violence and UCF-Crime datasets.
• Achieved 86.05% Average Precision (AP) on the XD-Violence dataset.
• Achieved 88.59% Area Under the Curve (AUC) on the UCF-Crime dataset.

Conclusions:

• The proposed text rewriting strategy significantly enhances semantic expression for VAD.
• The LGM-Mamba module effectively captures temporal dynamics and deepens feature fusion, boosting detection accuracy.
• TrCLIP-VAD represents a significant advancement in video anomaly detection by integrating advanced LLM and Mamba-based techniques.