Multimodal deep learning for entity relation extraction and spatiotemporal decision knowledge graph construction in...

Area of Science:

• Artificial Intelligence
• Computer Science
• Geophysics

Background:

• Earthquake emergency response faces challenges in integrating diverse data types.
• Effective decision support systems are crucial for timely and efficient rescue operations.

Purpose of the Study:

• To develop a novel framework integrating multimodal deep learning and spatiotemporal knowledge representation for earthquake emergency rescue.
• To enhance information processing and decision support capabilities in disaster scenarios.

Main Methods:

• Developed a Cross-modal Attention Fusion Network (CAFN) for heterogeneous data integration across text, visual, and spatiotemporal modalities.
• Employed a Transformer-based joint entity-relation extraction model for disaster information identification.
• Constructed a spatiotemporal knowledge graph with specialized reasoning for dynamic emergency scenarios.

Main Results:

• The entity-relation extraction model achieved an 89.0% F1 score, an 8.7% improvement over conventional methods.
• The decision support system demonstrated a 94.0% F1 score in resource allocation and route planning, a 23.7% improvement over rule-based systems.
• The framework effectively processed complex multimodal information for time-critical decisions.

Conclusions:

• The proposed framework significantly enhances information processing and decision support in earthquake emergency rescue.
• Multimodal deep learning and spatiotemporal knowledge representation are effective for improving disaster response.
• The system offers a robust solution for complex, dynamic emergency situations.