Limited performance of ChatGPT-4v and ChatGPT-4o in image-based core radiology cases

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging Analysis
• Radiology Education

Background:

• Large language models (LLMs) demonstrate capabilities in clinical reasoning and interpreting medical images.
• Recent advancements include LLMs with visual processing, enabling combined text and image analysis.

Purpose of the Study:

• To evaluate the performance of ChatGPT-4v and ChatGPT-4o in interpreting image-based multiple-choice questions from national radiology board examinations.
• To identify the limitations of these AI models in core radiologic scenarios.

Main Methods:

• A prospective study utilized 222 image-based multiple-choice questions from 2020-2024 national radiology board exams.
• ChatGPT-4v and ChatGPT-4o processed the questions, and their generated answers were compared against the official answer key.
• Accuracy was analyzed based on radiologic subspecialty and the inclusion of clinical information.

Main Results:

• ChatGPT-4o achieved a 59% success rate, and ChatGPT-4v achieved 54%, both below the board exam passing standard.
• No significant performance difference was observed between ChatGPT-4v and ChatGPT-4o (P=0.339).
• Both models performed better on questions with clinical information (ChatGPT-4v: 63.8%, ChatGPT-4o: 67.0%) compared to those without (ChatGPT-4v: 46.1%, ChatGPT-4o: 52.3%).

Conclusions:

• Current ChatGPT versions show potential as supplementary diagnostic aids but do not meet the accuracy required for board-level image interpretation.
• Performance variability across subspecialties indicates limitations and the need for further development.
• Extensive research is necessary before these AI tools can be safely integrated into clinical practice.