Computerized diagnostic decision support systems-Isabel Pro versus ChatGPT-4 part II.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support Systems

Background:

• Large language models (LLMs) like ChatGPT-4 are increasingly explored for clinical applications.
• Evaluating the diagnostic accuracy and reliability of LLMs is crucial before widespread adoption.
• Existing diagnostic decision support systems (DDSS) also have limitations that warrant comparison.

Purpose of the Study:

• To assess the diagnostic accuracy of ChatGPT-4 compared to the Isabel Pro DDSS.
• To investigate the impact of prompt engineering (Tree-of-Thought) and expert panel size on ChatGPT-4's performance.
• To evaluate the reproducibility and reference citation accuracy of ChatGPT-4 in a diagnostic context.

Main Methods:

• Utilized 201 cases from the New England Journal of Medicine for differential diagnosis generation.
• Employed statistical measures including Mean Reciprocal Rank (MRR), Recall at Rank, and Average Rank.
• Assessed reproducibility using r-squared calculations and evaluated citation accuracy for references and DOIs.

Main Results:

• ChatGPT-4 improved MRR and Recall at 10 but had fewer correct diagnoses and lower average rank.
• Revising the Isabel Pro differential improved Recall at 10 by 11%; a two-person expert panel yielded optimal results.
• Reproducibility runs showed poor consistency (r-squared 0.34-0.44), and reference accuracy was low (34.8% for citations, 37.8% for DOIs).

Conclusions:

• ChatGPT-4 demonstrates potential but faces significant challenges in diagnostic accuracy and reliability.
• Poor reproducibility and a high frequency of fabricated references limit its current clinical utility for diagnosis.
• Further development is required to address these critical issues before ChatGPT-4 can be safely integrated into clinical diagnostic workflows.