Modality-Agnostic Federated Learning With Adaptive Updates for Heterogeneous Medical Image Tasks

Federated learning (FL) enables collaborative model training across decentralized medical datasets while preserving data privacy. Its practical adoption remains limited due to data heterogeneity, specifically, differences in input imaging modality (e.g., CT or MRI) and client task (e.g., segmentation or classification) across participating institutions (clients). Such data heterogeneity poses significant challenges for jointly learning a unified global model that generalizes across clients with different input modality and task. To address this, we propose FedCMT, a modality-agnostic FL framework that adaptively aggregates heterogeneous client models. FedCMT supports flexible input modalities and diverse local tasks by incorporating group-wise adapters and personalized decoders that capture modality- and task-specific features. To enhance collaboration across clients, FedCMT employs a conflict-averse module that extracts modality-invariant representations and mitigates inter-client feature conflicts. FedCMT also integrates a global-to-local knowledge distillation mechanism to balance global consistency and local specialization. The proposed FedCMT maintains stability while fostering shared knowledge in diverse medical imaging modalities. We evaluate FedCMT on ten CT and MR datasets involving up to eight federated clients performing segmentation or classification tasks. Experimental results show that FedCMT consistently outperforms state-of-the-art FL baselines, yielding an average improvement of 4.76% over state-of-the-art methods and 4.01% over standalone training. These results demonstrate FedCMT as a promising adaptable FL for real-world medical image analysis.