A comprehensive framework for multi-modal hate speech detection in social media using deep learning.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision

Background:

• Online hate speech is increasingly multi-modal, combining text, images, audio, and video.
• Traditional single-modality detection systems struggle with complex, heterogeneous data streams.
• Nuanced hate speech, like memes and sarcastic videos, poses significant detection challenges.

Purpose of the Study:

• To propose a novel Multi-modal Hate Speech Detection Framework (MHSDF) for enhanced detection of online hate speech.
• To leverage hybrid deep learning models (CNNs and RNNs) for analyzing multi-modal data.
• To improve the accuracy, robustness, and interpretability of hate speech detection systems.

Main Methods:

• Developed a hybrid framework integrating Convolutional Neural Networks (CNNs) for spatial feature extraction and Recurrent Neural Networks (RNNs), specifically Long Short-Term Memory (LSTM), for sequential data.
• Utilized advanced word embeddings (Word2Vec, BERT) for textual analysis and Optical Character Recognition (OCR) for image text.
• Implemented attention mechanisms for effective fusion of features across modalities (text, image, audio, video).

Main Results:

• Achieved a detection accuracy ratio of 98.53%, robustness ratio of 97.64%, and performance ratio of 99.21%.
• Demonstrated significant improvements in interpretability (97.71%) and scalability (98.67%) compared to existing models.
• Attention-based explanations provided insights into multi-modal hate speech identification, enhancing transparency.

Conclusions:

• The proposed MHSDF effectively addresses the challenges of multi-modal hate speech detection.
• The hybrid CNN-RNN approach with attention mechanisms offers superior performance and interpretability.
• This framework provides a more transparent and robust solution for combating sophisticated online hate speech.