Applying Large Language Models to Assess Quality of Care: Monitoring ADHD Medication Side Effects
Yair Bannett1, Fatma Gunturkun2, Malvika Pillai3,4
1Division of Developmental-Behavioral Pediatrics, Stanford University School of Medicine, Stanford, California.
Insights
A large language model (LLM) accurately measured clinician adherence to ADHD medication guidelines by analyzing clinical notes. This technology offers scalable quality assessment for improving pediatric psychopharmacological care.
Area of Science:
- Artificial Intelligence in Healthcare
- Pediatric Psychopharmacology
- Clinical Quality Measurement
Background:
- Children with attention-deficit/hyperactivity disorder (ADHD) require careful medication management and monitoring for side effects.
- Assessing clinician adherence to practice guidelines for monitoring medication side effects in pediatric ADHD is crucial for quality care.
- Traditional methods of chart review for quality assessment are time-consuming and resource-intensive.
Purpose of the Study:
- To evaluate the accuracy of a large language model (LLM) in assessing clinician adherence to guidelines for monitoring ADHD medication side effects in children.
- To determine if an LLM can provide a scalable method for quality measurement in pediatric psychopharmacology.
Main Methods:
- A retrospective cohort study utilized electronic health records from 1201 children (aged 6-11 years) diagnosed with ADHD.
- An open-source LLM (LLaMA) was trained and deployed to analyze 15,628 clinical notes from ADHD-related encounters.
- Model performance was validated against manual medical record review, comparing its accuracy in identifying side effect inquiries.
Main Results:
- The LLM demonstrated high accuracy in identifying documentation of side effect inquiries (sensitivity 87.2%, specificity 86.3%, AUC 0.93).
- No significant model bias was found concerning patient sex or insurance status.
- Documented side effect inquiries were less frequent in telephone encounters (51.9%) compared to in-clinic/telehealth (73.0%) and varied between stimulant (61.4%) and nonstimulant (48.5%) prescriptions.
Conclusions:
- LLMs can be effectively deployed on diverse clinical notes, including telephone encounters, for scalable quality of care measurement.
- This approach facilitates the identification of areas for improvement in psychopharmacological medication management within primary care settings.
- Utilizing LLMs offers a promising avenue for enhancing the quality of care for children with ADHD.
Objective:
To assess the accuracy of a large language model (LLM) in measuring clinician adherence to practice guidelines for monitoring side effects after prescribing medications for children with attention-deficit/hyperactivity disorder (ADHD).
Methods:
Retrospective population-based cohort study of electronic health records. Cohort included children aged 6 to 11 years with ADHD diagnosis and 2 or more ADHD medication encounters (stimulants or nonstimulants prescribed) between 2015 and 2022 in a community-based primary health care network (n = 1201). To identify documentation of side effects inquiry, we trained, tested, and deployed an open-source LLM (LLaMA) on all clinical notes from ADHD-related encounters (ADHD diagnosis or ADHD medication prescription), including in-clinic/telehealth and telephone encounters (n = 15 628 notes). Model performance was assessed using holdout and deployment test sets, compared with manual medical record review.
Results:
The LLaMA model accurately classified notes that contained side effects inquiry (sensitivity = 87.2, specificity = 86.3, area under curve = 0.93 on holdout test set). Analyses revealed no model bias in relation to patient sex or insurance. Mean age (SD) at first prescription was 8.8 (1.6) years; characteristics were mostly similar across patients with and without documented side effects inquiry. Rates of documented side effects inquiry were lower for telephone encounters than for in-clinic/telehealth encounters (51.9% vs 73.0%, P < .001). Side effects inquiry was documented in 61.4% of encounters after stimulant prescriptions and 48.5% of encounters after nonstimulant prescriptions (P = .041).
Conclusions:
Deploying an LLM on a variable set of clinical notes, including telephone notes, offered scalable measurement of quality of care and uncovered opportunities to improve psychopharmacological medication management in primary care.
More Related Videos
13:09Using Brain Activation nir-HEG/Q-EEG and Execution Measures CPTs in a ADHD Assessment Protocol
Published on: April 1, 2018
05:48The Adventures of Fundi Intervention Based on the Cognitive and Emotional Processing in Attention Deficit Hyperactive Disorder Patients
Published on: June 12, 2020
Related Concept Videos
01:28Drug Therapy
Antianxiety Medications
01:26Modeling in Therapy
Participant Modeling
Participant modeling involves therapists demonstrating calm and effective behaviors in...
01:19Pharmacovigilance
This process, termed pharmacovigilance, aims to detect, evaluate, and minimize harmful effects related to medication use. The data collection for pharmacovigilance depends on spontaneous reporting systems, where healthcare professionals or patients voluntarily report suspected ADRs.
In some cases, there...
01:30Attention-Deficit/Hyperactivity Disorder
Diagnostic Criteria and Symptoms
To diagnose ADHD, symptoms must manifest before age 12 and be evident across multiple settings....