Evaluating ChatGPT-4V in chest CT diagnostics: a critical image interpretation assessment

Area of Science:

• Radiology
• Artificial Intelligence
• Medical Imaging

Background:

• Advancements in AI offer potential for augmenting radiological diagnostics.
• Cross-modal AI models present unique challenges in interpreting complex medical images like CT scans.

Purpose of the Study:

• To evaluate the diagnostic accuracy of ChatGPT-4V in interpreting chest CT slices for COVID-19, non-small cell lung cancer (NSCLC), and control cases.
• To assess the potential of ChatGPT-4V as an AI tool in radiological diagnostics.

Main Methods:

• Retrospective analysis of 60 chest CT scans (COVID-19, NSCLC, controls) using ChatGPT-4V.
• Selection of four representative CT slices per scan for AI interpretation.
• Comparison of AI diagnoses against gold standard diagnoses, validated by two radiologists.

Main Results:

• Overall diagnostic accuracy of ChatGPT-4V was 56.76%.
• Sensitivity and specificity varied across conditions: NSCLC (27.27% sensitivity, 60.47% specificity), COVID-19 (13.64% sensitivity, 64.29% specificity), controls (31.82% sensitivity, 95.24% specificity).
• Highest sensitivity (83.33%) observed when all lung lobes were involved; significant differences noted based on pathology location and lobar involvement.

Conclusions:

• ChatGPT-4V demonstrated variable diagnostic performance in chest CT interpretation.
• Significant challenges exist for cross-modal AI models in radiology, highlighting areas for improvement.
• Enhancing AI models is crucial for their dependable and broader application in medical diagnostics.