Concept-Aware Adaptive Multimodal Fusion With Knowledge Distillation for Medical Image Diagnosis

Multimodal medical image has emerged as a powerful tool for improving diagnostic accuracy by leveraging complementary information from diverse imaging modalities. However, existing methods often fail to account for disease-specific modality dependencies, leading to suboptimal performance in clinical scenarios. To address this limitation, we propose a Concept-Aware Adaptive Multimodal Fusion Knowledge Distillation (CA-AMD) framework that integrates hierarchical disease concepts with dynamic modality fusion. Our approach employs Chain-of-Thought (CoT) prompting to extract descriptive disease concepts from large language models (LLMs), which serve as semantic anchors for disentangling and enhancing multimodal features. The framework dynamically selects specialized fusion teachers through a gating network, enabling adaptive fusion tailored to disease attributes. Knowledge distillation is then applied to transfer concept-aware representations to a lightweight student network, ensuring efficient deployment without information leakage. Extensive experiments on three publicly available datasets (Derm7pt, MMC-AMD, and Harvard30k-Glaucoma) demonstrate that CA-AMD surpasses state-of-the-art methods, achieving absolute improvements of 5.37%, 3.91%, and 6.93% in Cohen's Kappa, respectively. The results highlight the framework's ability to model disease-modality relationships and enhance diagnostic robustness.