Enhancing early detection of cognitive decline in the elderly: a comparative study utilizing large language models in...

Area of Science:

• Artificial Intelligence in Healthcare
• Clinical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show promise in healthcare but require evaluation for detecting specific clinical conditions in electronic health records (EHRs).
• This study assesses LLMs for identifying cognitive decline indicators in clinical notes, comparing their performance and error patterns against traditional models.
• Understanding these differences informs strategies for improving LLM performance in clinical settings.

Purpose of the Study:

• To evaluate the effectiveness of large language models (LLMs) in detecting cognitive decline from electronic health record (EHR) clinical notes.
• To compare the error profiles of LLMs (GPT-4, Llama 2) with traditional machine learning models.
• To develop an optimal LLM-based method for identifying cognitive decline and assess the performance of an ensemble model.

Main Methods:

• Analysis of clinical notes from Mass General Brigham (2019 diagnosis of mild cognitive impairment).
• Development and comparison of prompts for GPT-4 and Llama 2 using various approaches (hard prompting, retrieval augmented generation).
• Creation of an ensemble model combining LLMs and traditional models (hierarchical attention network, XGBoost) using a majority vote, evaluated with confusion-matrix-based scores.

Main Results:

• GPT-4 showed superior accuracy and efficiency over Llama 2 but did not surpass traditional models individually.
• The ensemble model significantly outperformed individual models across all metrics (p < 0.01), achieving 90.2% precision, 94.2% recall, and 92.1% F1-score.
• The ensemble model substantially improved precision (from 70%-79% to >90%), with only 3.2% mutual errors across all models, indicating diverse error profiles.

Conclusions:

• LLMs and traditional models trained on local EHR data have distinct error profiles.
• An ensemble approach combining these models is complementary and enhances diagnostic performance for cognitive decline detection.
• Future research should explore integrating LLMs with localized models and domain-specific knowledge for task-specific performance enhancement.