MMDOK: A Multi-modal and Multi-scale Disease-oriented Fusion Framework with Kolmogorov-Arnold Networks

Area of Science:

• Artificial Intelligence
• Medical Imaging Analysis
• Representation Learning

Background:

• Medical image analysis often relies on single modalities, overlooking the rich information in associated textual reports.
• Heterogeneity between medical images and reports presents challenges for effective data fusion in diagnostic models.

Purpose of the Study:

• To develop an advanced fusion framework for integrating multi-modal medical data (images and reports) to improve diagnostic accuracy.
• To leverage multi-scale features and adaptively fuse information from diverse sources for disease-oriented analysis.

Main Methods:

• Proposed a Multi-modal and Multi-scale Disease-oriented fusion framework with Kolmogorov-Arnold Networks (MMDOK).
• Employed Bidirectional Cross-Attention (BCA) and Cross-Modal Clustering (CMC) modules for enhanced feature extraction across semantic scales.
• Integrated a Disease-Oriented Attention (DOA) module to reduce cross-modal redundancy and assign modality-specific weights.
• Utilized Kolmogorov-Arnold Network layers for efficient, nonlinear feature extraction.

Main Results:

• Achieved 94.27% accuracy on a lung disease dataset with text supervision, significantly outperforming alternatives.
• Demonstrated strong performance on a private submucosal tumors dataset, with 87.63% accuracy.
• Showcased improved generalization to out-of-distribution data through the Cross-Modal Clustering module.

Conclusions:

• The MMDOK framework offers a robust approach for multi-modal medical data fusion, enhancing diagnostic capabilities.
• Adaptive fusion strategies and advanced network architectures significantly improve accuracy and generalization in medical image analysis.
• This work highlights the potential of integrating diverse data sources and advanced deep learning techniques for superior medical diagnostics.