Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare

Background:

• Medicine generates diverse data types: images, time-series, audio, text, video, and omics.
• Current large language models (LLMs) primarily process unimodal (text) data, limiting their clinical application.
• Integrating diverse medical data is crucial for comprehensive patient understanding and decision-making.

Purpose of the Study:

• To present a practical, solution-oriented perspective on multimodal large language models (M-LLMs) in medicine.
• To explore foundational principles, applications, challenges, and future directions of M-LLMs in healthcare.
• To offer a unified vision for M-LLMs, guiding research and implementation.

Main Methods:

• Literature review and analysis of M-LLM capabilities in the medical domain.
• Exploration of foundational principles and technical aspects of M-LLMs.
• Identification and discussion of current and potential applications, ethical considerations, and future research avenues.

Main Results:

• M-LLMs offer a framework to integrate diverse medical data modalities, overcoming limitations of unimodal LLMs.
• Identified key applications ranging from diagnostics to personalized treatment planning.
• Highlighted significant technical and ethical challenges requiring further investigation.

Conclusions:

• M-LLMs represent a paradigm shift towards integrated, multimodal data-driven medical practice.
• This work provides a comprehensive framework to guide future research and practical implementation of M-LLMs in healthcare.
• Further development of M-LLMs is expected to drive innovation in next-generation medical AI systems.