Benchmarking Foundation Models with Multimodal Public Electronic Health Records

Area of Science:

• Artificial Intelligence in Medicine
• Biomedical Informatics
• Machine Learning for Healthcare

Background:

• Foundation models offer versatile processing of diverse electronic health records (EHRs).
• Evaluating their performance, fairness, and interpretability is crucial for clinical adoption.
• Standardized benchmarks are needed for reproducible AI research in healthcare.

Purpose of the Study:

• To benchmark foundation models (unimodal and multimodal) using the MIMIC-IV database.
• To assess model performance, fairness, and interpretability.
• To guide the development of trustworthy multimodal AI for clinical applications.

Main Methods:

• Developed a standardized data processing pipeline for heterogeneous EHR data.
• Evaluated twelve foundation models (unimodal, multimodal, domain-specific, general-purpose).
• Assessed performance, fairness, and interpretability metrics.

Main Results:

• Multimodal foundation models generally improve predictive performance without increasing bias.
• Domain-specific fine-tuning is effective for unimodal models but not multimodal.
• Current large vision-language models show limited task generalizability in medical contexts.

Conclusions:

• Multimodal foundation models show promise for EHR analysis, enhancing performance and fairness.
• Further research is needed to improve the generalizability of large vision-language models for medical tasks.
• This benchmark supports the creation of reliable AI tools for clinical practice.