LLM-Driven Medical Report Generation via Communication-Efficient Heterogeneous Federated Learning

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Machine Learning

Background:

• Large Language Models (LLMs) show promise for Medical Report Generation (MRG).
• Developing LLM-driven MRG models requires extensive multi-center medical image-report data.
• Data centralization is hindered by privacy regulations, limiting LLM adoption in MRG.

Purpose of the Study:

• Introduce FedMRG, a novel framework for privacy-preserving, multi-center LLM development in MRG.
• Address communication efficiency challenges in federated LLM training.
• Mitigate dual data heterogeneity in medical imaging and reporting styles.

Main Methods:

• Federated Learning (FL) framework for privacy-preserving MRG model development.
• Low-rank factorization to reduce communication overhead in federated LLM tuning.
• Client-aware contrastive learning and diagnosis-driven prompts for encoder heterogeneity.
• Dual-adapter mutual boosting mechanism in the decoder for reporting style variations.

Main Results:

• FedMRG demonstrates effective LLM training for MRG in a federated setting.
• The framework successfully addresses communication efficiency and data heterogeneity.
• Evaluations confirm the generalizability and adaptability of FedMRG on a dedicated benchmark.

Conclusions:

• FedMRG enables collaborative, privacy-preserving development of LLM-driven MRG models.
• The framework enhances communication efficiency and handles multi-modal data heterogeneity.
• FedMRG facilitates the generation of clinically accurate medical reports across multiple centers.