Comparative diagnostic capability of large language models in neurosurgery

Area of Science:

• Artificial Intelligence
• Neurosurgery
• Medical Diagnostics

Background:

• Large language models (LLMs) like ChatGPT, Gemini, and Copilot show promise in clinical applications.
• Specialized neurosurgery LLMs, such as AtlasGPT, are emerging.
• Comparative analysis of neurosurgical diagnostic capabilities of these LLMs is lacking.

Purpose of the Study:

• To evaluate and compare the diagnostic accuracy of various LLMs in neurosurgery.
• To assess the performance of OpenAI's ChatGPT-3.5 and ChatGPT-4, Google's Gemini, Microsoft Copilot, and AtlasGPT.
• To identify factors influencing LLM diagnostic performance in neurosurgical pathologies.

Main Methods:

• 148 neurosurgical clinical vignettes from a textbook were used.
• LLMs were prompted to diagnose based on vignette history, with imaging data included where possible.
• Diagnoses were compared against the textbook for accuracy, with errors categorized.

Main Results:

• ChatGPT-4 achieved the highest accuracy (74%), followed by AtlasGPT (63%), ChatGPT-3.5 (53%), Copilot (48%), and Gemini (36%).
• ChatGPT-4 significantly outperformed other models (p = 0.005).
• Common errors involved misinterpreting or omitting key information, particularly imaging data, despite logical reasoning.

Conclusions:

• ChatGPT-4 provided the most accurate neurosurgical diagnoses in this study.
• Incorporating imaging capabilities and data enhances LLM diagnostic accuracy.
• While LLMs show potential for neurosurgical condition assessment, critical evaluation and precise prompting by clinicians are essential.