Multimodal Data Hybrid Fusion and Natural Language Processing for Clinical Prediction Models

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Data Fusion Techniques

Background:

• Clinical prediction models often rely on limited data sources, impacting their accuracy.
• Integrating diverse data types like electronic health records (EHRs) and clinical notes presents challenges.
• Existing multimodal fusion methods may not fully capture complex patient information.

Purpose of the Study:

• To develop and evaluate a novel hybrid fusion method for enhancing clinical prediction models.
• To integrate structured EHR data, unstructured clinical notes, and National Electronic Injury Surveillance System (NEISS) data.
• To improve the accuracy and robustness of predictive models through multimodal data fusion.

Main Methods:

• Proposed a comprehensive framework integrating multimodal data sources.
• Developed a hybrid fusion method incorporating a pre-trained language model (RoBERTa).
• Fused unstructured clinical text with structured EHR data and NEISS data for comprehensive patient representation.

Main Results:

• The hybrid fusion approach significantly outperformed traditional fusion frameworks and unimodal models.
• The RoBERTa-based hybrid fusion system achieved 75.00% accuracy for Top 1 injury prediction.
• The system achieved 93.54% accuracy for Top 3 injury prediction.

Conclusions:

• Integrating NLP with multimodal data fusion enhances clinical prediction model performance.
• The proposed approach improves accuracy and robustness by leveraging rich clinical text and structured EHR data.
• This method has potential applications in clinical decision support, personalized medicine, and evidence-based healthcare.