AtlasGPT: a language model grounded in neurosurgery with domain-specific data and document retrieval

Area of Science:

• Artificial Intelligence in Medicine
• Neurosurgical Knowledge Management
• Large Language Models (LLMs)

Background:

• General large language models (LLMs) show potential in medical exams but lack subspecialty expertise and robustness.
• Assessing LLM performance in specialized medical fields like neurosurgery is crucial for safe and effective application.
• The ability of LLMs to generate accurate explanations and resist adversarial attacks needs further investigation.

Purpose of the Study:

• To introduce AtlasGPT, a subspecialty-focused LLM for neurosurgery.
• To evaluate AtlasGPT's performance on a neurosurgery question bank and under adversarial conditions.
• To assess the quality of explanations generated by AtlasGPT.

Main Methods:

• AtlasGPT was developed using GPT-4 fine-tuning and retrieval-augmented generation from neurosurgical data.
• Performance comparison with GPT-4 and Gemini Advanced on a 149-question neurosurgery exam.
• Adversarial testing to evaluate robustness against misinformation, with explanations rated by 15 neurosurgeons.

Main Results:

• AtlasGPT achieved 96% accuracy, outperforming Gemini Advanced (93%) and GPT-4 (88%) on neurosurgery questions.
• In adversarial testing, AtlasGPT was misled by misinformation only 14% of the time, significantly better than GPT-4 (44%) and Gemini Advanced (68%).
• Neurosurgical experts rated AtlasGPT's explanations as superior in comprehensiveness, relevance, and referencing, with no harmful content detected.

Conclusions:

• Subspecialty-focused LLMs like AtlasGPT can surpass general models in accuracy and robustness.
• AtlasGPT demonstrates potential for improving medical knowledge, clinical decision-making, and educational resources in neurosurgery.
• Domain-specific LLMs offer a promising avenue for advancing complex medical fields.