ChatGPT-Generated Differential Diagnosis Lists for Complex Case-Derived Clinical Vignettes: Diagnostic Accuracy...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support

Background:

• The diagnostic accuracy of AI chatbots, including ChatGPT, for complex clinical vignettes is not well-established.
• General internal medicine (GIM) case reports present unique diagnostic challenges.

Purpose of the Study:

• To evaluate the diagnostic accuracy of ChatGPT-3.5 and ChatGPT-4 for complex clinical vignettes.
• To compare AI-generated differential diagnoses with those from human physicians.

Main Methods:

• Utilized PubMed-sourced GIM case reports from Dokkyo Medical University Hospital.
• Generated top 10 differential diagnoses using ChatGPT-3.5 and ChatGPT-4 with clinical vignettes.
• Compared AI-generated lists with differential diagnoses created by three independent GIM physicians.

Main Results:

• ChatGPT-4 achieved 83% accuracy in top 10 differential diagnoses, compared to 73% for ChatGPT-3.5.
• ChatGPT-4's accuracy (83%) was comparable to physicians' (75%) for top 10 differential diagnoses.
• AI diagnostic accuracy was not significantly affected by publication date or open access status.

Conclusions:

• ChatGPT-3.5 and ChatGPT-4 demonstrate potential in generating accurate differential diagnoses for complex GIM cases.
• ChatGPT-4's high accuracy in top 10 and top 5 lists suggests its utility as a supplementary tool for physicians.
• Further research with diverse datasets is needed to fully understand AI's role in clinical decision-making.