Visual-textual integration in LLMs for medical diagnosis: A preliminary quantitative analysis

Area of Science:

• Medical diagnostics
• Artificial intelligence in healthcare
• Natural Language Processing

Background:

• Visual data is crucial for medical diagnoses.
• Evaluating multimodal Large Language Models (LLMs) for diagnostic tasks integrating text and images.

Purpose of the Study:

• Assess the diagnostic performance of GPT-4o and Claude Sonnet 3.5.
• Compare LLM performance with and without integrated visual data against human physicians.

Main Methods:

• 120 clinical vignettes (text-only and with images) were used.
• Images were sourced from OPENi and NEJM challenges, ensuring they were novel to LLMs.
• Three primary care physicians served as a benchmark for diagnostic accuracy.

Main Results:

• LLMs outperformed physicians in text-only scenarios (GPT-4o: 70.8%, Claude Sonnet 3.5: 59.5%, Physicians: 39.5%).
• Image integration improved all participants' accuracy, with physicians showing the largest gain (Physicians: 78.8%).
• LLMs adjusted their reasoning in 45-60% of cases when images were provided.

Conclusions:

• Multimodal LLMs demonstrate strong diagnostic accuracy, sometimes surpassing physicians even without visual input.
• While images enhance LLM performance, the relative gain is less than that observed in human physicians.
• Further advancements in LLM visual data processing are needed to match human-like performance gains from medical images.