Clinician-in-the-loop screening saturation: predicting annotation yield for efficient EHR review

Area of Science:

• Health Informatics
• Artificial Intelligence in Medicine
• Clinical Data Analysis

Background:

• Manual chart review of Electronic Health Records (EHRs) is time-consuming and costly, posing a challenge for retrospective studies, especially for rare diseases.
• Automated Natural Language Processing (NLP) tools improve efficiency but require updated data for accurate performance predictions.
• A need exists for models that predict the remaining data yield during chart review to optimize resource allocation.

Purpose of the Study:

• To develop and validate a regression-based 'screening-saturation' model to predict residual data yield in EHR chart reviews.
• To assess the model's performance in identifying true positives at various score thresholds.
• To enable data-driven staffing decisions for clinical workflows.

Main Methods:

• Trained four predictive models (linear, polynomial, support-vector regressions, and a neural network) on 2013 EHR data for pediatric status epilepticus.
• Validated model generalizability and performance using 2020 EHR data.
• Evaluated the proportion of true positives identified below specific score thresholds.

Main Results:

• The polynomial regression model demonstrated robust generalizability and interpretability across temporal data shifts.
• A 20% yield threshold identified 78.1% of true positives by reviewing only 14.6% of records (1,118/7,636).
• This approach achieved an 85.4% reduction in manual annotations, saving 6,518 annotations.

Conclusions:

• The proposed screening-saturation model offers a scalable and model-agnostic framework for optimizing chart review processes.
• It effectively translates AI scores into actionable staffing decisions within clinical workflows.
• The model is adaptable to various medical domains requiring efficient chart review and integrates with clinician-in-the-loop tools.