Large language models in neurosurgery: a systematic review and meta-analysis

Area of Science:

• Neurosurgery
• Artificial Intelligence
• Medical Informatics

Background:

• Large Language Models (LLMs) are increasingly recognized for their potential impact on neurosurgery.
• Systematic examination of LLM capabilities across neurosurgical tasks is lacking.
• Challenges include LLM terminology and ensuring replicability.

Purpose of the Study:

• To identify key LLMs used in neurosurgery literature.
• To establish reporting guidelines for enhancing research replicability.
• To highlight progress in neurosurgical applications of LLMs.

Main Methods:

• Searched PubMed and Google Scholar for LLMs and neurosurgery-related terms.
• Reviewed 51 articles for publication year, application area, LLM used, and reporting of prompts and performance.
• Assessed reproducibility, hallucination measures, and performance metrics.

Main Results:

• Fifty-one articles were included, focusing on text generation, exam question answering, and clinical decision support.
• GPT-3.5 (58.8%) and GPT-4 (39.2%) were the most utilized LLMs.
• Most studies (84.3%) used LLMs without advanced customization, indicating a gap in sophisticated application.

Conclusions:

• LLMs demonstrate significant capabilities for neurosurgery, with potential for transformative impact.
• Current research often focuses on basic applications and lacks detailed reporting, hindering reproducibility.
• Standardized reporting, addressing LLM stochasticity, and employing advanced methods are crucial for future progress.