Clinician-in-the-loop screening saturation: predicting annotation yield for efficient EHR review

Labor- and cost-intensive manual chart review of Electronic Health Records (EHRs) remains a major bottleneck in retrospective studies, particularly when rare-disease cohorts require high specificity. Automated Natural Language Processing (NLP) rankers help, yet when trained on dated data, they leave teams guessing how long to keep reviewing charts. Therefore, this study presents a regression-based "screening-saturation" model that predicts residual yield at every point along the ranked list.

Using a previously validated Support Vector Machine (SVM) that ranks notes for pediatric status epilepticus, the authors first trained four predictive models: linear, polynomial, and support-vector regressions plus a lightweight neural net, on notes from 2013. Then, these models were tested on data from 2020. The target was to identify the proportion of true positives (ESE or RSE) expected below any score threshold.

Polynomial regression offered the best balance of generalizability and interpretability, demonstrating a strong predictive performance even under temporal data shifts. On 2020 notes, a 20% yield threshold captured 78.1% of positives after reviewing 14.6% of records (1,118/7,636), which is equivalent to an 85.4% reduction in manual annotations (i.e., 6,518 fewer).

The proposed scalable, model-agnostic framework turns AI scores into actionable staffing decisions in clinical workflows. This screening-saturation model integrates with clinician-in-the-loop (CITL) tools and readily adapts across medical domains requiring lean chart review.