MTMedFormer: multi-task vision transformer for medical imaging with federated learning

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Computer Vision

Background:

• Deep learning significantly enhances medical imaging analysis but faces challenges with large dataset requirements and data privacy.
• Existing multi-task learning (MTL) models struggle with global feature contextualization, while federated learning (FL) methods face difficulties in aggregating stable feature maps.

Purpose of the Study:

• To propose MTMedFormer, a novel transformer-based model for multi-task medical imaging.
• To address data scarcity and privacy issues in training diagnostic models.
• To improve the aggregation of multi-task imaging models in a federated setting.

Main Methods:

• Developed MTMedFormer, a transformer architecture with a shared encoder for task-agnostic features and task-specific decoders.
• Integrated multi-task learning with a hybrid loss function for synergistic task learning.
• Introduced a novel Bayesian federation method for aggregating multi-task imaging models.

Main Results:

• MTMedFormer demonstrated superior performance compared to single-task and traditional MTL models on mammogram and pneumonia datasets.
• The proposed Bayesian federation method achieved better results in image segmentation than conventional aggregation techniques.
• The model effectively learns distinct diagnostic tasks synergistically.

Conclusions:

• MTMedFormer offers a robust solution for multi-task medical image analysis, overcoming limitations of previous approaches.
• The developed Bayesian federation method enhances collaborative model training without compromising data privacy.
• This work advances the application of deep learning in medical imaging by enabling efficient and private collaborative learning.