ChatGPT-4.0 and Medical Students: A Recognition-Gated Comparative Evaluation on Image-Based Medical Examinations

Area of Science:

• Medical Education Technology
• Artificial Intelligence in Medicine
• Computer Vision in Healthcare

Background:

• Artificial intelligence (AI), especially vision-capable large language models (LLMs) like ChatGPT 4.0, is increasingly influencing medical education.
• The efficacy of these AI tools in visually intensive, image-based medical disciplines requires further investigation.

Purpose of the Study:

• To compare the performance of ChatGPT 4.0 against undergraduate medical students on standardized, image-based multiple-choice questions.
• To assess AI performance across different medical domains: Anatomy, Pathology, and Pediatrics.

Main Methods:

• A cohort study administered standardized image-based exams to medical students (second, third, and fifth years).
• The same questions were posed to ChatGPT 4.0 using a two-step protocol; questions with unrecognized images were excluded.
• Statistical analysis involved paired t-tests or Wilcoxon signed-rank tests, with subgroup analyses focusing on questions with a discrimination index ≥ 0.1.

Main Results:

• ChatGPT 4.0 significantly underperformed students in Anatomy (mean difference = -0.387) but outperformed them in Pediatrics (mean difference = +0.174).
• No Pathology questions were eligible for comparative analysis due to ChatGPT 4.0's failure in image recognition.
• In a global analysis scoring image-recognition failures as incorrect, ChatGPT 4.0 achieved 17.3% accuracy in Anatomy, 84.9% in Pediatrics, and 0% in Pathology.

Conclusions:

• ChatGPT 4.0 exhibits significant variability in visual reasoning across medical disciplines.
• Critical evaluation and multimodal integration are essential before adopting AI in image-dependent medical education.