Humans vs. large language models in neurology board examination: performance, limitations, and reference reliability

Area of Science:

• Neurology
• Artificial Intelligence
• Medical Education

Background:

• Large language models (LLMs) are increasingly used in various fields, including medicine.
• Evaluating their performance in specialized medical domains like neurology is crucial.

Purpose of the Study:

• To assess the accuracy and reference reliability of three leading LLMs (ChatGPT Plus, Gemini Advanced, Microsoft Copilot Pro) on neurology board examination questions.
• To compare LLM performance against human examinee performance.

Main Methods:

• 803 multiple-choice questions from Turkish National Neurology Board Examinations (2015-2024) were used.
• LLMs were prompted to provide answers and supporting references.
• Performance was analyzed by neurological subspecialty, question type, and visual content, and compared to human examinee data.
• References for incorrect answers were evaluated by neurologists.

Main Results:

• LLMs achieved high accuracy rates (85-87%), significantly outperforming the human average of 65%.
• Accuracy was comparable across LLMs and did not vary by question type or subspecialty.
• LLMs performed better on non-visual questions; no advantage was seen on visual items.
• Significant issues with reference reliability were found, including insufficient citations and fabricated references.

Conclusions:

• LLMs demonstrate strong performance on neurology board exams, exceeding human examinee accuracy.
• Visually based questions and reference quality present limitations for LLMs.
• LLMs should be utilized as supplementary educational tools under expert supervision, not as independent authorities.