Solving gaps in clinical reasoning is the cure to neurophobia in artificial intelligence

This is an observational study that assesses the clinical reasoning of an artificial intelligence chatbot using previously published cases. The primary objective of the study is to assess the unique challenges neurologic cases pose to machine large language models (LLMs) in clinical practice in the context of human errors in clinical reasoning.

This observational study tested the accuracy of GPT-4o, an artificial intelligence-powered chatbot, in generating a differential diagnosis for a series of 29 cases. The cases were presented in two formats, one as the full protocol with all detailed information, and a second clinical syndrome with information distilled via clinical reasoning into core features of pace and localization. The primary outcome was comparison of GPT-4o diagnostic accuracy when given all clinical information versus the distilled clinical syndrome.

GPT-4o performed equally as well or better when provided the clinical syndrome versus the full protocol for every of the 29 cases provided. The overall accuracy improved from approximately 76 % to 97 % when given the clinical syndrome, and the overall rank of the correct diagnosis within the differential generated was equal or better for every clinical syndrome compared to the full case protocol.

Our study demonstrates that clinical reasoning remains a major barrier in diagnostic accuracy in artificial intelligence just as it is mirrored in human trainees. Focusing training not just on knowledge, but on clinical reasoning, has the potential to improve performance of machine LLMs and learners alike dramatically.