Dedicated AI Expert System vs Generative AI With Large Language Model for Clinical Diagnoses

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Clinical Decision Support

Background:

• Large language models (LLMs) have emerged as powerful tools, but their diagnostic capabilities compared to traditional systems are not well-established.
• Diagnostic decision support systems (DDSSs) are established tools, yet their performance against newer LLMs on novel clinical data requires evaluation.

Purpose of the Study:

• To compare the diagnostic performance of two leading LLMs (ChatGPT-4 and Gemini 1.5) against a traditional DDSS (DXplain).
• To evaluate system performance with and without the inclusion of laboratory test results using unpublished clinical cases.

Main Methods:

• A diagnostic study involving 36 unpublished general medicine cases from three academic medical centers.
• Physician reviewers identified relevant clinical findings; blinded data entry into LLMs and DDSS was performed with and without laboratory data.
• Performance was assessed by the presence and rank of the correct diagnosis within the top 25 differential diagnoses generated by each system.

Main Results:

• Without laboratory data, the DDSS showed a trend towards higher diagnostic accuracy (56%) than LLM1 (42%) and LLM2 (39%), though not statistically significant.
• Inclusion of laboratory test results improved diagnostic accuracy for all systems, with the DDSS achieving 72%, LLM1 64%, and LLM2 58%.
• All systems successfully listed the correct diagnosis within the top 25 for most cases when laboratory data was provided.

Conclusions:

• Current LLMs and a traditional DDSS demonstrate comparable, though not statistically significant, performance on unpublished clinical cases when laboratory data is omitted.
• The inclusion of laboratory test results substantially enhances the diagnostic accuracy of both LLMs and DDSSs.
• A hybrid approach integrating LLM's linguistic strengths with DDSS's deterministic capabilities may offer synergistic benefits for clinical decision support.