Large Language Models in Neurological Practice: Real-World Study.

Area of Science:

• Medical Diagnostics
• Artificial Intelligence in Healthcare
• Neurology

Background:

• Large language models (LLMs) like ChatGPT and Gemini are being explored for medical diagnostic applications, particularly in neurology.
• The real-world applicability and diagnostic accuracy of these freely available LLMs in clinical workflows remain insufficiently evaluated.

Purpose of the Study:

• To assess the diagnostic accuracy of untrained, freely available LLMs (ChatGPT and Gemini) compared to neurologists.
• To evaluate the appropriateness of clinical recommendations and diagnostic test suggestions made by LLMs in real-world neurology cases.

Main Methods:

• An experimental evaluation comparing LLM diagnostic performance against clinical neurologists using 28 anonymized real-world neurology cases.
• Simulated first-visit scenarios based on patient records from a hospital neurology department.
• Primary outcome: diagnostic accuracy (concordance with discharge diagnoses); Secondary outcomes: appropriateness of test recommendations, interrater agreement, and prompting needs.

Main Results:

• Neurologists achieved 75% diagnostic accuracy, surpassing ChatGPT (54%) and Gemini (46%).
• LLMs exhibited limitations in nuanced reasoning and over-recommended diagnostic tests (17%-25% of cases).
• Complex cases required additional prompting for LLMs, and moderate agreement (κ=0.47) was observed among human raters.

Conclusions:

• Freely available, untrained LLMs currently lack the necessary depth for independent clinical decision-making in neurology.
• The study underscores the need for rigorous validation of AI tools before integration into clinical workflows.
• Future research should focus on enhancing LLM capabilities and developing robust evaluation methodologies for real-world neurological practice.