Optical Flow-Aware-Based Multi-Modal Fusion Network for Violence Detection

Area of Science:

• Computer Vision
• Artificial Intelligence
• Multimedia Security

Background:

• Violence detection in video is crucial for security applications.
• Existing methods often fail to fully utilize multi-modal vision and audio information, limiting accuracy.
• Optical flow changes correlate with video violence, suggesting its utility.

Purpose of the Study:

• To propose an optical flow-aware multi-modal fusion network (OAMFN) for improved violence detection.
• To leverage multi-modal features and optical flow information for more accurate video analysis.
• To enhance the integration of visual (RGB, optical flow) and audio data.

Main Methods:

• Developed an optical flow-aware-based multi-modal fusion network (OAMFN).
• Employed three fusion strategies: concatenation of RGB/audio features, integration of optical flow with RGB/audio, and cross-modal fusion with attention.
• Introduced an optical flow-aware score fusion strategy for combining multi-modal features.

Main Results:

• The OAMFN achieved significant improvements in average precision (AP) on the XD-Violence dataset.
• Offline detection APs were 83.09% and 1.4% higher than state-of-the-art methods.
• Online detection APs were 78.09% and 4.42% higher than state-of-the-art methods.

Conclusions:

• The proposed OAMFN effectively integrates multi-modal information and optical flow for superior violence detection.
• The optical flow-aware approach enhances the network's ability to discern violent content.
• This method offers a substantial advancement for both online and offline video security systems.