Unveiling GPT-4V's hidden challenges behind high accuracy on USMLE questions: Observational Study.

Area of Science:

• Artificial Intelligence in Medicine
• Medical Image Analysis
• Large Language Models

Background:

• Recent AI advancements like GPT-4 show promise in text-based medical exams.
• The capability of AI models to interpret medical images remains largely unexplored.
• Evaluating AI in multimodal clinical scenarios is crucial for assessing readiness.

Purpose of the Study:

• To evaluate GPT-4 Vision (GPT-4V), GPT-4, and GPT-3.5 Turbo on image-based medical questions.
• To assess GPT-4V's performance, interpretability, and limitations in understanding clinical images.
• To identify potential flaws and readiness for clinical integration of multimodal AI.

Main Methods:

• A cross-sectional study using 227 multiple-choice questions with images from USMLE, DRQCE, and AMBOSS.
• Compared accuracy of GPT-4V against GPT-4 and GPT-3.5 Turbo.
• Evaluated explanation quality using human preference and qualitative metrics; simulated curbside consultations.

Main Results:

• GPT-4V achieved high accuracy (73.1%-88.9%) on image-based questions, outperforming GPT-4 and GPT-3.5 Turbo.
• Correct answers had expert-level explanations, but incorrect answers showed poor quality, including image misunderstandings.
• GPT-4V required specific prompts for complex cases and showed reduced error rates with expert assistance.

Conclusions:

• GPT-4V demonstrates significant potential for medical image interpretation assessments.
• Shortcomings in explanation quality and image interpretation for incorrect answers require attention.
• Comprehensive evaluations beyond multiple-choice questions are necessary before clinical deployment.