Unified Multi-Modal Diagnostic Framework With Reconstruction Pre-Training and Heterogeneity-Combat Tuning

Area of Science:

• Artificial Intelligence in Medicine
• Medical Informatics
• Computer-Aided Diagnosis

Background:

• Medical multi-modal pre-training shows potential for computer-aided diagnosis using large unlabeled datasets.
• Existing masked autoencoder methods lack high-level semantic information and struggle with data and modality heterogeneity.

Purpose of the Study:

• To propose a Unified Medical Multi-modal Diagnostic (UMD) framework to enhance medical multi-modal pre-training and downstream tuning.
• To address distribution and modality heterogeneity challenges in medical diagnostic tasks.

Main Methods:

• Developed a Multi-level Reconstruction Pre-training (MR-Pretrain) strategy for feature-level and data-level reconstruction.
• Introduced a heterogeneity-combat downstream tuning strategy with Task-oriented Distribution Calibration (TD-Calib) and Gradient-guided Modality Coordination (GM-Coord).

Main Results:

• The UMD framework demonstrated superior performance across five public medical datasets.
• The proposed pre-training and tuning strategies effectively captured semantic information and managed data heterogeneity.

Conclusions:

• The UMD framework significantly outperforms existing methods in medical multi-modal diagnostic tasks.
• The tailored strategies enhance the transferability of pre-trained knowledge to downstream applications, improving diagnostic accuracy.