[Research progress on electronic health records multimodal data fusion based on deep learning]

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Deep Learning

Background:

• Deep learning-based multimodal learning is rapidly advancing, particularly in AI-generated content like image-text tasks.
• Electronic health records (EHRs) are crucial digital data generated during medical activities.
• Integrating multimodal EHR data with deep learning offers potential for comprehensive analysis in healthcare.

Purpose of the Study:

• To introduce deep learning-based multimodal data fusion methods and trends.
• To summarize and compare fusion techniques for structured EHRs with other medical data (images, text).
• To discuss challenges and future directions in multimodal medical data fusion.

Main Methods:

• Reviewing existing literature on deep learning for multimodal medical data fusion.
• Analyzing clinical applications, sample sizes, and fusion methodologies.
• Identifying key deep learning approaches: pre-trained models and attention mechanisms.

Main Results:

• Deep learning enables comprehensive analysis of multimodal EHR data for improved diagnosis and patient intervention.
• Effective fusion methods involve selecting appropriate pre-trained models and utilizing attention mechanisms.
• Current research focuses on integrating structured EHRs with imaging and textual data.

Conclusions:

• Multimodal fusion of EHRs using deep learning can significantly aid medical professionals.
• Further research is needed in modeling, evaluation, and application of these fusion techniques.
• The goal is to develop models that effectively utilize diverse medical data modalities for enhanced healthcare outcomes.