Clinical Manifestations.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Medical Informatics

Background:

• Alzheimer's Disease and Related Dementias (ADRD) pose public health challenges, necessitating accurate cognitive impairment (CI) diagnosis.
• Electronic health records (EHRs) contain valuable CI data within unstructured clinical notes.
• Large language models (LLMs) like GPT-4o offer potential for analyzing EHR data for CI identification and staging.

Purpose of the Study:

• To develop and evaluate a GPT-4o-powered framework for CI staging and automated Clinical Dementia Rating (CDR) scoring using EHR data.
• To compare the performance of the GPT-4o framework against traditional NLP and embedding-based models for CI classification.

Main Methods:

• A GPT-4o framework was developed for CI staging, incorporating data querying, feature extraction, and classification on 1002 Medicare patients.
• GPT-4o generated multi-note summaries and performed ordinal CI classification with confidence levels.
• Automated CDR scoring was achieved using structured answer templates, retrieval-augmented generation (RAG), and CDR domain counts with confidence levels.

Main Results:

• The GPT-4o framework demonstrated high accuracy in CI staging (weighted Cohen's kappa = 0.95, Spearman correlation = 0.93).
• Performance was superior to traditional feature extraction and embedding-based models, especially for high-confidence predictions.
• Automated CDR scoring using domain counts achieved high accuracy (weighted Cohen's kappa = 0.83).

Conclusions:

• The GPT-4o framework shows significant potential for automated chart review in dementia diagnosis, outperforming existing models.
• Clinical deployment necessitates a human-in-the-loop approach to mitigate misclassification risks and ensure safety.
• This hybrid AI-human model balances AI's diagnostic capabilities with interpretability and risk management.