XGeM: A multi-prompt foundation model for multimodal medical data generation

Area of Science:

• Artificial Intelligence in Medical Imaging
• Generative AI for Healthcare
• Multimodal Data Synthesis

Background:

• AI in medical imaging offers potential but faces hurdles like data scarcity, privacy, and multimodal integration.
• Current generative models often produce unimodal or unidirectional synthetic data, lacking clinical consistency.
• Existing methods struggle with joint synthesis of multiple medical data modalities.

Purpose of the Study:

• To introduce XGeM, a large-scale multimodal generative model for flexible, any-to-any medical data synthesis.
• To overcome limitations of existing generative models in handling multiple data types simultaneously.
• To provide a foundation model for addressing critical challenges in medical data.

Main Methods:

• Developed XGeM, a 6.77-billion-parameter multimodal generative model.
• Constructed a shared latent space using contrastive learning.
• Implemented a novel Multi-Prompt Training strategy for conditioning on arbitrary input modality subsets.

Main Results:

• XGeM demonstrated superior performance compared to five competitors on the MIMIC-CXR dataset.
• Expert radiologists confirmed the clinical relevance and realism of XGeM-generated data via a Visual Turing Test.
• XGeM effectively supported medical data anonymization, class imbalance, and data scarcity challenges.

Conclusions:

• XGeM offers a powerful solution for multimodal medical data synthesis, preserving clinical consistency.
• The model's flexibility and ability to generate coherent multimodal outputs address key limitations in current AI.
• XGeM serves as a foundational model with significant potential for advancing medical AI applications.