CMAP-Fusion: A cross-modal feature selection and model pruning framework for laboratory and imaging data

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Data Fusion

Background:

• Cross-modal fusion of medical imaging and laboratory data is crucial for disease diagnosis.
• Existing methods face challenges like modal heterogeneity, feature redundancy, and efficiency imbalance, limiting precision and clinical adaptability.

Purpose of the Study:

• To propose the Cross-Modal Alignment-Pruning Fusion (CMAP-Fusion) model for optimized medical cross-modal fusion.
• To address limitations in precision, efficiency, and generalization of current fusion techniques.

Main Methods:

• Utilized Vision Transformer (ViT-B/16) for imaging feature extraction and dimension alignment.
• Implemented the SmartTrim dynamic pruning module to screen key features and reduce redundancy.
• Employed the Cross-Modal Transformer (CMT) to mine deep associations between dual modalities.

Main Results:

• Achieved high accuracies: 95.3% (COVID-19), 89.7% (ISIC Skin Cancer), 93.6% (ChestX-ray14).
• Improved accuracy by 3.1%-4.1% over optimal baselines.
• Reduced parameters by 44.2% and computational complexity by over 43%.

Conclusions:

• CMAP-Fusion synergistically optimizes precision, efficiency, and generalization for medical cross-modal fusion.
• The model offers an efficient solution, outperforming baselines in accuracy, efficiency, and feature representation.
• Future work includes expanding to multi-source modalities, multi-disease scenarios, and clinical validation.