Testing the Ability and Limitations of ChatGPT to Generate Differential Diagnoses from Transcribed Radiologic...

Area of Science:

• Artificial Intelligence in Medicine
• Radiology and Medical Imaging
• Natural Language Processing

Background:

• Growing interest in ChatGPT's medical applications necessitates rigorous evaluation.
• Assessing AI tools like ChatGPT is crucial for safe and effective clinical integration.
• Understanding limitations of AI in diagnostic processes is paramount.

Purpose of the Study:

• To evaluate the accuracy, reliability, and repeatability of ChatGPT's differential diagnoses from radiology findings.
• To compare the performance of GPT-3.5 and GPT-4 algorithms in generating differential diagnoses.
• To identify limitations, such as hallucination and repeatability issues, in AI-driven diagnostic suggestions.

Main Methods:

• 339 radiology cases from a textbook series were used, covering diverse modalities and pathologies.
• Standardized prompts were fed into ChatGPT (GPT-3.5 and GPT-4) between April 3 and June 1, 2023.
• Responses were compared against textbook ground truth for accuracy, reliability (hallucinations), and repeatability (test-retest).

Main Results:

• GPT-4 achieved higher overall accuracy (66.1%) compared to GPT-3.5 (53.7%) for final diagnoses (P < .001).
• GPT-4 produced fewer fabricated references (14.3%) and false statements (4.7%) than GPT-3.5 (39.9% and 16.2%, respectively; P < .001).
• Repeatability varied significantly, with average pairwise agreement for top diagnoses ranging from 23%-49% across models.

Conclusions:

• ChatGPT (GPT-4) shows improved accuracy and reliability over GPT-3.5 for radiologic differential diagnoses.
• Prompting for a single diagnosis and using the GPT-4 model yielded the best results.
• Despite improvements, significant issues with repeatability and hallucination persist, requiring further investigation for clinical use.