Large Language Models and Large Multimodal Models in Medical Imaging: A Primer for Physicians

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging Analysis
• Health Informatics

Background:

• Large language models (LLMs) show significant potential for health care applications.
• The evolution towards large multimodal models (LMMs) enables processing of both text and images, expanding capabilities.
• Physician understanding of LLMs and LMMs is vital for effective and responsible implementation in clinical practice.

Purpose of the Study:

• To explain the fundamental concepts behind LLM development and application.
• To detail the technical process of creating LMMs.
• To discuss current and future use cases of LLMs and LMMs in medical imaging.

Main Methods:

• Explanation of core LLM concepts: token embeddings, transformer networks, self-supervised pretraining, and fine-tuning.
• Description of the technical pipeline for developing LMMs.
• Review of existing and potential applications in the medical imaging domain.

Main Results:

• LLMs and LMMs offer promising applications in medical imaging.
• Understanding the underlying technology empowers physicians to utilize AI tools more effectively.
• The development of LMMs facilitates integrated text and image data analysis.

Conclusions:

• LLMs and LMMs are set to significantly impact health care, particularly in medical imaging.
• Educating physicians on LLM principles is essential for responsible adoption and development.
• Further exploration of LMM use cases in medical imaging is warranted.