neuroGPT-X: toward a clinic-ready large language model

Area of Science:

• Neurosurgery
• Artificial Intelligence
• Medical Informatics

Background:

• Vestibular schwannoma management presents complex challenges, often lacking definitive evidence-based consensus.
• Neurosurgical experts are crucial for decision-making, but human memory limitations can impact point-of-care information retrieval.
• Large language models (LLMs) offer potential for augmenting clinical decision support systems.

Purpose of the Study:

• To evaluate the performance of a context-enriched LLM against neurosurgical experts in answering questions about vestibular schwannoma management.
• To develop and assess a chat-based platform (neuroGPT-X) for real-time, reliable clinical information delivery with citations and memory capabilities.

Main Methods:

• A dataset was created via web scraping.
• Eight international neurosurgical experts developed questions and evaluated LLM responses (blinded).
• The neuroGPT-X platform was developed and evaluated against 103 consensus statements on vestibular schwannoma care.

Main Results:

• Context-enriched LLMs performed comparably or better than experts in accuracy, coherence, relevance, and thoroughness.
• LLMs provided significantly faster responses than human experts (p < 0.01).
• The context-enriched LLM aligned with 95% of consensus statements, though experts raised concerns about reliability in nuanced cases.

Conclusions:

• Context-enriched LLMs show significant promise as point-of-care medical resources, especially in areas with limited consensus.
• The developed platform, neuroGPT-X, aims to enhance clinical support by providing accurate, cited information and mitigating memory limitations.
• This work serves as a foundation for expanding LLM applications across medical specialties for expert-level dialogue and evidence-based support.