Multi-modal deep learning framework for damage detection in social media posts.

Area of Science:

• Artificial Intelligence
• Computer Science
• Crisis Informatics

Background:

• Effective crisis management requires rapid identification of affected individuals and damage assessment, especially with limited information.
• Traditional emergency systems face challenges in reachability and managing high request volumes during disasters.
• Social media platforms are vital for real-time information dissemination and rescue efforts when standard communication fails.

Purpose of the Study:

• To develop an automated framework for detecting damage in social media posts during emergencies.
• To enhance the efficiency and accuracy of crisis response by processing large volumes of social media data.
• To improve the identification of disaster-affected areas and survivor conditions through content analysis.

Main Methods:

• A hybrid framework combining Bidirectional Encoder Representations from Transformers (BERT) for text analysis and convolutional neural network (CNN) blocks for image processing.
• Utilizing BERT-based networks to understand the semantic meaning of text in social media posts.
• Employing multiple CNN blocks to analyze visual information and detect damage within images.

Main Results:

• The proposed framework demonstrated superior performance compared to existing methods.
• Achieved high accuracy, recall, and F1 scores in detecting damage from social media content.
• Effectively processed and analyzed combined textual and visual data for crisis information extraction.

Conclusions:

• The integrated BERT and CNN approach offers a powerful solution for automated damage detection in social media during crises.
• This method has the potential to significantly improve emergency response times and effectiveness.
• Future enhancements could include multimodal analysis, incorporating audio data to further boost prediction efficiency.