Comparing the Neuropsychology Knowledge Base of Publicly Available Large Language Models.

Area of Science:

• Artificial Intelligence
• Neuropsychology
• Medical Education

Background:

• Large language models (LLMs) are increasingly integrated into various professional fields.
• Assessing the knowledge base of LLMs in specialized domains like neuropsychology is crucial for understanding their potential and limitations.

Purpose of the Study:

• To evaluate the neuropsychology knowledge of open-access LLMs.
• To identify potential applications and limitations of LLMs in neuropsychology.

Main Methods:

• 600 multiple-choice neuropsychology questions were used to test OpenAI (GPT-3.5, GPT-4, o3-mini-high) and Google (Gemini 1.0, 2.0 FTE) models.
• Two testing trials were conducted, followed by statistical analysis including paired-samples t-tests and binomial logit generalized linear mixed-effects models (GLMMs).
• Missed questions by top models were thematically analyzed.

Main Results:

• OpenAI's o3-mini-high achieved the highest accuracy (90.3% in T2), followed by Gemini 2.0 FTE (88.7%).
• LLMs generally overestimated their accuracy by an average of 15.8%.
• Chain-of-thought reasoning models outperformed older models, but inaccuracies were noted in testing interpretation, diagnostic reasoning, and neuroanatomy.

Conclusions:

• Chain-of-thought reasoning LLMs show potential utility in neuropsychology education, research, and practice.
• Persistent weaknesses in specific neuropsychology content areas and overconfidence in incorrect answers necessitate careful interpretation of LLM outputs.