Multimodal large language models for medical image diagnosis: Challenges and opportunities

Andrew Zhang ¹, Eric Zhao ¹, Ruirui Wang ¹

⁰Center of Applied Science and Technology, AlphaPi Institution, Los Angeles, CA 90017, USA.

Journal of Biomedical Informatics +

|

August 20, 2025

View abstract on PubMed

Summary

Multimodal large language models (MLLMs) show promise in radiology but face adoption hurdles. Future research should focus on transparency, privacy, and standardization to improve AI in medical imaging and patient care.

Area Of Science

Radiology
Artificial Intelligence
Medical Imaging

Background

Artificial intelligence (AI) has enhanced diagnostic accuracy and workflow in radiology.
Multimodal large language models (MLLMs) integrate natural language processing (NLP) and computer vision for advanced medical image analysis.

Purpose Of The Study

To explore the potential of MLLMs in revolutionizing medical image analysis.
To identify challenges hindering the clinical adoption of MLLMs in radiology.
To outline future research priorities for enhancing MLLM performance and reliability.

Main Methods

Review of current advancements and limitations of MLLMs in radiology.
Analysis of challenges including data quality, interpretability, regulatory compliance (e.g., GDPR), computational demands, and generalizability.
Identification of key areas for future research and development.

Main Results

MLLMs offer significant potential for improving radiology diagnostics and efficiency.
Clinical adoption is currently limited by data quality, interpretability, ethical concerns, and generalizability issues.
Addressing these challenges is crucial for realizing the full potential of MLLMs.

Conclusions

Overcoming barriers in data quality, transparency, and standardization is essential for MLLM integration.
Future research should prioritize federated learning for data privacy and robust evaluation frameworks.
Successful implementation of MLLMs can transform radiology, aiding clinical decisions and improving patient outcomes.

Keywords:

AI Generalizability Computer vision Large Language Model Medical image diagnosis Multimodal fusion strategy Natural Language Processing (NLP) Radiology AI