Can ChatGPT outperform a neurosurgical trainee? A prospective comparative study

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Surgical Training

Background:

• The integration of artificial intelligence (AI), particularly large language models (LLMs), into healthcare is rapidly advancing.
• Assessing the capabilities of AI in high-stakes medical assessments is crucial for understanding its potential and limitations.

Purpose of the Study:

• To compare the performance of ChatGPT against human neurosurgical applicants in a simulated national selection interview.
• To evaluate the potential of AI and LLMs in healthcare settings and inform their future integration.

Main Methods:

• A prospective comparative study involving eight human participants and ChatGPT answering neurosurgical national selection-style interview questions.
• Interviews were conducted online, anonymised, and scored by experienced neurosurgical consultants using established national selection criteria.

Main Results:

• ChatGPT's overall interview performance was lower than six of the eight human participants.
• ChatGPT did not achieve a mean score higher than any individual who secured training positions.
• Factual inaccuracies and deviations in structure and style contributed to ChatGPT's underperformance.

Conclusions:

• Large language models (LLMs) demonstrate significant potential for integration within the healthcare sector.
• Further advancements are necessary to overcome current limitations and challenges in AI performance for medical applications.
• While AI has not yet surpassed human capabilities in this context, collaborative human-AI systems offer a promising future for healthcare innovation.