Specialized Large Language Model Outperforms Neurologists at Complex Diagnosis in Blinded Case-Based Evaluation.

Area of Science:

• Neurology
• Artificial Intelligence
• Medical Diagnostics

Background:

• Large Language Models (LLMs) show broad applicability but struggle in specialized fields like neurology.
• Evaluating the diagnostic capability and trustworthiness of a specialized LLM in neurology is crucial.
• This study compares AI performance against human neurologists in simulated neurological diagnostic scenarios.

Purpose of the Study:

• To assess the diagnostic performance of a specialized LLM in complex neurological cases.
• To compare the accuracy and efficiency of the AI system against practicing neurologists.
• To evaluate the trustworthiness and verifiability of AI-generated diagnostic information.

Main Methods:

• GPT-4 Turbo LLM deployed via Neura AI infrastructure with dual-database architecture.
• A curated neurological corpus was used for training and evaluation.
• 13 neurologists and the AI system evaluated 5 clinical scenarios, providing differential and definitive diagnoses.

Main Results:

• AI achieved a significantly higher normalized score (86.17%) than neurologists (55.11%, p < 0.001).
• AI demonstrated higher accuracy in both differential (85% vs 46.15%) and final diagnoses (88.24% vs 70.93%).
• AI responded in under 30 seconds, significantly faster than neurologists' average of 9 minutes, with all references verified.

Conclusions:

• The specialized LLM exhibited superior diagnostic performance compared to practicing neurologists in complex neurological challenges.
• LLMs, when integrated with curated knowledge bases, can achieve domain-specific relevance in complex clinical disciplines.
• This suggests AI's potential as an efficient and accurate asset in clinical neurological practice.