Research on Multimodal Fusion of Temporal Electronic Medical Records

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Machine Learning

Background:

• Deep learning in electronic medical record (EMR) research increasingly utilizes diverse data modalities.
• Integrating diverse data types, especially within time series EMR data, remains a significant challenge.
• Existing methods often struggle with the complexity of multimodal, temporal clinical data.

Purpose of the Study:

• To develop and evaluate a novel multimodal fusion approach for EMR data.
• To effectively integrate temporal and non-temporal clinical notes with tabular data.
• To enhance predictive accuracy for patient outcomes using a comprehensive EMR data representation.

Main Methods:

• A multimodal fusion model was developed, combining static and time series note and table data.
• Temporal data was preprocessed, segmented, and processed using a long short-term memory (LSTM) network.
• Multimodal attention gates and an attention-backtracking module were employed for fused representation and capturing temporal dependencies.

Main Results:

• The proposed fusion model demonstrated superior predictive performance compared to baseline and recent models like Crossformer.
• The attention-backtracking module was crucial for performance, indicating the importance of capturing long-range temporal dependencies.
• The model effectively integrated four data modalities and handled irregular time series and lengthy clinical texts.

Conclusions:

• The developed multimodal fusion method offers an effective approach for integrating diverse EMR data, including complex temporal and textual information.
• The model shows significant potential for improving predictive tasks in healthcare by leveraging comprehensive patient data.
• This method is expected to see broader application in multimodal medical data representation and analysis.