Using Machine Learning to Capture Quality Metrics from Natural Language: A Case Study of Diabetic Eye Exams.

Area of Science:

• Health Informatics
• Natural Language Processing
• Clinical Quality Measurement

Background:

• Value-based payment models increase reliance on electronic health records (EHRs) for quality measure reporting.
• This necessitates efficient methods for extracting quality data from clinical documentation.

Purpose of the Study:

• To apply text mining and natural language processing (NLP) to identify timely completion of diabetic eye exams (DEEs) from clinician notes.
• To evaluate machine learning models for classifying DEE completion as an electronic clinical quality measure (eCQM).

Main Methods:

• Utilized text mining and NLP on 26,203 unique clinician notes.
• Compared logistic regression and support vector machine (SVM) models, with and without synthetic minority over-sampling technique (SMOTE), for DEE classification.
• Evaluated models based on precision, recall, sensitivity, and f1-score.

Main Results:

• SVM models demonstrated high performance, with the best precision at 0.96 and recall at 0.85.
• Applied SVM models to a large dataset, identifying a significant percentage of patients with completed DEEs.
• Highlighted the potential utility of these models for quality measure reporting.

Conclusions:

• Automated extraction of DEE data using SVM models can reliably identify completed exams, reducing manual review burden.
• This approach enables capturing eCQM data from routine clinical practice, enhancing quality reporting without increasing clinician workload.
• The methodology can be extended to extract information for other quality measures from clinical notes.