Estimating depression severity in narrative clinical notes using large language models.

Area of Science:

• Clinical Informatics
• Natural Language Processing
• Mental Health Research

Background:

• Current depression treatment guidelines advocate for measurement-based care utilizing patient-reported outcome measures (PROMs).
• The impact of PROMs on the quality of narrative clinical documentation has not been extensively studied.
• Understanding this impact is crucial for optimizing electronic health record (EHR) utilization in mental healthcare.

Purpose of the Study:

• To evaluate the ability of a foundational large language model (LLM) to estimate depression severity from clinical notes after censoring patient-reported outcome scores.
• To assess the correlation between LLM-estimated and actual depression scores (PHQ-9).
• To examine the LLM's predictive performance for moderate to severe depressive symptoms.

Main Methods:

• A dataset of 15,000 outpatient clinical notes with corresponding 9-item Patient Health Questionnaire (PHQ-9) scores was analyzed.
• PHQ-9 scores were censored from the notes, and a foundational LLM (gpt4o-08-06) estimated depression severity within a HIPAA-compliant environment.
• Statistical analysis included correlation estimation and predictive performance assessment for moderate or greater depressive symptoms.

Main Results:

• The LLM-estimated PHQ-9 scores showed a modest correlation with actual scores (r² = 0.264).
• The positive predictive value (PPV) for identifying moderate or greater depression was 0.309.
• LLM performance was consistent across demographic subgroups, with minor variations observed across race, ethnicity, and sex.

Conclusions:

• Foundational LLMs demonstrate poor but consistent performance in inferring depression severity from clinical notes when patient-reported data is absent.
• The findings suggest that the integration of PROMs might inadvertently lead to a reduction in the detailed documentation of psychiatric symptoms.
• Further research is needed to balance the benefits of PROMs with the preservation of rich clinical narrative in EHRs.