Multi-modal fusion model for Time-Varying medical Data: Addressing Long-Term dependencies and memory challenges in...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Data Fusion Techniques

Background:

• Patient hospitalization generates multi-modal time-varying data reflecting disease progression.
• Current fusion models struggle with long-term states and often neglect textual data like ward notes.
• There's a need for models that can effectively process long sequences and long texts.

Purpose of the Study:

• To develop an effective medical multi-modal time-varying data fusion model.
• To extract features from long sequences and long texts.
• To capture long-term dependencies in patient data.

Main Methods:

• Proposed the Medical Multi-modal Fusion for Long-term Dependencies (MMF-LD) model.
• Introduced a Progressive Multi-modal Fusion (PMF) strategy to prevent information loss in time-varying text fusion.
• Integrated attention mechanisms with Long Short-Term Storage memory (LSTsM) and Temporal Convolutional Networks (TCN) for enhanced feature extraction.

Main Results:

• MMF-LD demonstrated superior performance over existing models on acute myocardial infarction (AMI) and stroke datasets.
• Achieved high AUROC (up to 0.965) and AUPRC (up to 0.675) for mortality and length-of-stay predictions.
• Ablation studies confirmed the contributions of PMF, LSTsM, and TCN modules.

Conclusions:

• The MMF-LD architecture effectively addresses the challenge of incorporating long textual information in time series fusion.
• The model exhibits stable performance across multiple datasets and prediction tasks.
• It shows significant strength in capturing long-term dependencies crucial for patient care.