MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling.

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Biomedical Data Science

Background:

• Multimodal electronic health record (EHR) data offer comprehensive patient insights but integrating them for clinical prediction is data-intensive and challenging.
• Existing methods struggle with the heterogeneity and volume of multimodal EHR data, limiting predictive accuracy.

Purpose of the Study:

• To introduce a novel architecture, Mixture-of-Multimodal-Agents (MoMA), for effective clinical prediction using multimodal EHR data.
• To leverage multiple large language model (LLM) agents to overcome data integration challenges in multimodal EHR analysis.

Main Methods:

• MoMA utilizes specialized LLM agents to convert non-textual data (images, labs) into structured textual summaries.
• An aggregator LLM agent combines these summaries with clinical notes into a unified multimodal summary.
• A predictor LLM agent generates clinical predictions based on the unified summary.

Main Results:

• MoMA demonstrated superior performance compared to existing methods across three distinct clinical prediction tasks.
• The architecture was evaluated using various combinations of data modalities and prediction settings on private datasets.
• MoMA achieved enhanced accuracy and flexibility in multimodal EHR data analysis.

Conclusions:

• The MoMA architecture effectively integrates diverse multimodal EHR data for clinical prediction.
• MoMA offers a flexible and accurate approach to leveraging LLM agents for complex healthcare data challenges.
• This novel method shows significant potential for advancing clinical prediction modeling using rich EHR information.