Enhancing Early Detection of Cognitive Decline in the Elderly: A Comparative Study Utilizing Large Language Models in...

Area of Science:

• Artificial Intelligence in Healthcare
• Clinical Informatics

Background:

• Evaluating large language models (LLMs) for identifying cognitive decline in electronic health records (EHRs).
• Comparing LLM performance against traditional models using real-world clinical notes.
• Addressing the under-explored effectiveness of LLMs in specific clinical condition detection.

Approach:

• Analyzed clinical notes from Mass General Brigham (2019-2023) for patients aged 50+ diagnosed with mild cognitive impairment.
• Developed and optimized prompts for GPT-4 and Llama 2 LLMs on HIPAA-compliant platforms.
• Created an ensemble model combining LLMs with hierarchical attention network and XGBoost for enhanced diagnostic performance.

Key Points:

• GPT-4 showed better accuracy and efficiency than Llama 2 but did not surpass traditional models individually.
• The ensemble model significantly improved precision (to >90%) and overall performance (92.2% F1-score).
• Diverse error profiles were observed across models, with minimal mutual errors, highlighting complementary strengths.

Conclusions:

• LLMs and traditional models exhibit distinct error patterns when analyzing EHR data for cognitive decline.
• Ensemble modeling effectively leverages complementary strengths, enhancing diagnostic accuracy.
• Future work should focus on integrating LLMs with localized models and domain-specific medical knowledge.