Limited performance of ChatGPT-4v and ChatGPT-4o in image-based core radiology cases

Large language models such as ChatGPT have shown potential in clinical reasoning and radiologic interpretation. Recent versions with image-analysing capabilities allow for combined visual and textual processing. This study aims to assess the performance and limitations of ChatGPT-4v and ChatGPT-4o in interpreting image-based multiple-choice questions from official national radiology board examinations, which are designed to reflect core radiologic scenarios.

This prospective study used 222 image-based multiple-choice official questions from a national radiology board examinations administered between 2020 and 2024. Questions were entered into ChatGPT-4v and ChatGPT-4o; Generated answers were compared to the official answer key. Accuracy was further analysed by radiologic subspecialty and the presence or absence of clinical information.

ChatGPT-4o achieved a 59 % (130/222) success rate, while ChatGPT-4v achieved a 54 % (119/222), with both models underperforming relative to the board exam passing standard. No significant difference was found between the two versions (two-tailed P-value = 0.339). Analysis by subspecialty revealed that ChatGPT-4v had a similar success rate across all fields (p = 0.330), whereas the success rate of ChatGPT-4o varied significantly (p = 0.0009). Both models achieved significantly higher success rates on questions that included clinical information. ChatGPT-4v: 63.8 % (60/94) vs. 46.1 % (59/128), p = 0.0099; ChatGPT-4o: 67.0 % (63/94) vs. 52.3 % (67/128), p = 0.0384.

ChatGPT shows potential as a supportive diagnostic tool, but its accuracy remains below the standard required for board-level image interpretation. The variability across subspecialties highlights current limitations and underscores the need for further research before safe clinical integration.