Clinical Manifestations.

Area of Science:

• Neuroscience
• Artificial Intelligence
• Cognitive Psychology

Background:

• Clinical neuropsychological assessments, especially narrative recall tests, are challenging for real-time scoring.
• Post-hoc scoring improves accuracy but is time-consuming.
• Large Language Models (LLMs) offer potential for reliable and consistent scoring of verbal responses.

Purpose of the Study:

• To evaluate the efficacy of ChatGPT-4o in scoring the Craft Story 21 assessment.
• To compare LLM scoring with human rater performance.
• To explore the potential of LLMs as a reliable and cost-effective alternative for neuropsychological assessment scoring.

Main Methods:

• 28 participants (14 MCI, 14 CN) completed the Craft Story 21 assessment (immediate and delayed recall).
• ChatGPT-4o was provided with human-corrected transcripts and two sets of scoring criteria (standard and novel).
• Few-shot and chain-of-thought prompting strategies were employed.

Main Results:

• LLM scoring showed moderate correlations with human raters for both immediate (r=0.654) and delayed recall (r=0.655).
• Interrater reliability was comparable between LLM (ICC2k=0.580) and human raters (ICC2k=0.574).
• LLM scoring demonstrated superior discrimination of cognitive status for immediate recall (AUC=0.765) compared to human scoring (AUC=0.638).

Conclusions:

• LLMs show potential as a reliable, cost-effective alternative to human scoring for research purposes.
• Current LLM technology is premature for clinical settings due to hallucinations and score variability.
• Advancements in LLMs may lead to viable clinical applications in the near future.