Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Multimodal foundation models exploit text to make medical image predictions.

Thomas A Buckley1, James A Diao1,2, Cam N Srivastava1

  • 1Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA.

Nature Communications
|June 12, 2026
PubMed
Summary
This summary is machine-generated.

Related Concept Videos

Mouse Models of Cancer Study02:43

Mouse Models of Cancer Study

Mice have long served as models for studying human biology and pathology because of their phylogenetic and physiological similarity with humans. They are also easy to maintain and breed in the laboratory, and hence, many inbred strains are now available for research. Studies on mice have contributed immeasurably to our understanding of cancer biology.
The development of transgenic, knockout, and knock-in mice has led to an exponential increase in their use as model organisms in research,...

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Immediate vs delayed concussion reporting: role of concussion knowledge, attitudes, and symptoms.

Brain injury·2026
Same author

Screening for Missed Opportunities for Diagnosis in the ED Using eTriggers and Large Language Models.

JAMA network open·2026
Same author

Large reasoning models as thinking machines for medicine.

Nature biomedical engineering·2026
Same author

BRIDGE: benchmarking large language models for understanding real-world clinical practice texts.

Nature biomedical engineering·2026
Same author

Towards Conversational AI for Disease Management.

Nature·2026
Same author

Artificial intelligence to revolutionize surgical decision-making is still just around the corner.

Surgery·2026

Multimodal AI models for medical imaging rely heavily on text, not images. Misleading text drastically reduces accuracy, even when images alone are sufficient for diagnosis.

Area of Science:

  • Artificial Intelligence
  • Medical Imaging
  • Clinical Decision Support

Background:

  • Multimodal foundation models (MFMs) show promise in medical image interpretation but their modality integration is unclear.
  • Understanding how MFMs prioritize text and image data is crucial for reliable diagnostic reasoning.

Purpose of the Study:

  • To evaluate the performance of 8 multimodal foundation models in medical image interpretation.
  • To investigate how these models integrate and prioritize image and text data for diagnostic accuracy.

Main Methods:

  • Evaluation of 8 proprietary and open-source MFMs on 1090 multimodal medical cases.
  • Analysis of model predictions based on varying amounts of informative and misleading text.
  • Physician evaluation of model performance on long-form cases with and without image integration.

Related Experiment Videos

Main Results:

  • Model accuracy is monotonically dependent on the informativeness of the accompanying text.
  • Misleading text significantly degrades diagnostic performance, reducing accuracy from 84% to 28% in some cases.
  • Adding images to highly informative text did not improve, and sometimes decreased, performance in physician evaluations (e.g., GPT-4V).

Conclusions:

  • Multimodal AI models' accuracy in medical diagnostics is predominantly driven by textual information.
  • While potentially useful, the reliance on text presents a double-edged sword, susceptible to errors in clinical vignettes.
  • Further research is needed to improve the balanced integration of image and text data in multimodal medical AI.