Evaluating AI Models: Performance Validation Using Formal Multiple-Choice Questions in Neuropsychology.

Area of Science:

• Neuroscience
• Artificial Intelligence
• Medical Education

Background:

• Accessible and high-quality education is vital for the advancement of clinical neuropsychology.
• Barriers to board certification in clinical neuropsychology include time constraints and knowledge gaps.
• Artificial Intelligence (AI) language models offer potential solutions for educational challenges.

Purpose of the Study:

• To evaluate the performance of advanced AI language models, GPT-3.5 and GPT-4.0, on clinical neuropsychology board certification-like questions.
• To identify specific areas where AI models struggle in neuropsychological assessment and interpretation.

Main Methods:

• Performance evaluation of GPT-3.5 and GPT-4.0 using 300 American Board of Professional Psychology in Clinical Neuropsychology-like questions.
• Comparative analysis of accuracy rates between GPT-3.5 and GPT-4.0.
• Thematic analysis of incorrectly answered questions, particularly within the 'Assessment' category.

Main Results:

• GPT-4.0 achieved a significantly higher accuracy rate (80.0%) compared to GPT-3.5 (65.7%).
• In the 'Assessment' category, GPT-4.0's accuracy was 73.4% versus GPT-3.5's 58.6% (p=0.012).
• Thematic analysis revealed key AI knowledge gaps in 'Neurodegenerative Diseases' and 'Neuropsychological Testing and Interpretation'.

Conclusions:

• Advanced AI language models, particularly GPT-4.0, demonstrate considerable potential in supporting clinical neuropsychology education and board certification preparation.
• AI models require further development to address specific knowledge gaps in complex areas like neurodegenerative diseases and test interpretation.
• AI tools could help overcome educational barriers by providing accessible and specialized knowledge resources.