Evaluating ChatGPT-4's Diagnostic Accuracy: Impact of Visual Data Integration.

Area of Science:

• Medical Artificial Intelligence
• Clinical Diagnostics
• Multimodal AI Systems

Background:

• Multimodal generative AI, like ChatGPT-4 with vision (ChatGPT-4V), integrates text and visual data for advanced analysis.
• The potential impact of image data integration on AI diagnostic accuracy in healthcare remains largely unexplored.

Purpose of the Study:

• To evaluate the effect of incorporating image data on ChatGPT-4's diagnostic accuracy.
• To compare the diagnostic performance of ChatGPT-4V (text + image) against ChatGPT-4 (text-only).

Main Methods:

• 363 case reports from the American Journal of Case Reports (Jan 2022-Mar 2023) were analyzed after exclusions.
• Diagnostic accuracy was assessed by comparing AI-generated differential diagnoses with final diagnoses from case reports.
• Two physicians independently reviewed accuracy, with a third resolving discrepancies.

Main Results:

• ChatGPT-4V did not significantly improve the inclusion of final diagnoses in top 10 differential lists (85.1% vs 87.9%).
• The text-only ChatGPT-4 outperformed ChatGPT-4V in correctly identifying the top diagnosis (55.9% vs 44.4%).
• AI self-reports indicated image data contributed only 30% weight to differential diagnoses in over half of cases.

Conclusions:

• Current ChatGPT-4V systems primarily rely on textual information, underutilizing visual data's diagnostic potential.
• Further advancements are needed for multimodal AI to effectively integrate and leverage clinical image data for enhanced diagnostic performance.
• Improved multimodal data integration in AI systems could significantly benefit patient care through more accurate diagnostic insights.