Text Dialogue Analysis for Primary Screening of Mild Cognitive Impairment: Development and Validation Study

Area of Science:

• Neurology
• Artificial Intelligence
• Computational Linguistics

Background:

• Specialized AI models show success in diagnosing cognitive impairment.
• The efficacy of large language models (LLMs) like ChatGPT for this task using text alone is not well-established.

Purpose of the Study:

• To evaluate ChatGPT's performance in the primary screening of mild cognitive impairment (MCI).
• To standardize prompt design elements for optimal LLM performance in cognitive screening.

Main Methods:

• Utilized text dialogues from 174 participants (70% training, 30% testing).
• Developed a 5-part prompt incorporating vocabulary, syntax, and semantics.
• Included indicators for MCI severity: tip-of-the-tongue phenomenon, complex idea difficulty, and memory issues.

Main Results:

• The GPT-4 model achieved high performance on the training set (Sensitivity: 0.86, Specificity: 0.95, AUC: 0.91).
• On the test set, performance remained strong (Sensitivity: 0.77, Specificity: 0.83, AUC: 0.80).
• Key indicators significantly discriminated between MCI and cognitively normal participants (P<.001).

Conclusions:

• ChatGPT demonstrates effectiveness for the primary screening of potential MCI.
• Standardizing prompts, particularly by clinicians, can further enhance model performance.
• ChatGPT serves as a screening tool and is not a substitute for professional clinical diagnosis.