Semi-supervised ROC analysis for reliable and streamlined evaluation of phenotyping algorithms.

Area of Science:

• Biomedical Informatics
• Machine Learning in Healthcare
• Translational Research

Background:

• High-throughput phenotyping using electronic health records (EHRs) is crucial for translational research.
• Estimating and evaluating phenotyping algorithms (PAs) requires extensive medical supervision, posing a significant bottleneck.
• Existing weakly-supervised learning methods lack reliable evaluation strategies for PAs with limited labeled data.

Purpose of the Study:

• To introduce a novel semi-supervised approach, ssROC, for estimating receiver operating characteristic (ROC) parameters of PAs.
• To address the challenge of evaluating PA predictive performance with a very small proportion of labeled data.
• To enable precise estimation of PA performance metrics like sensitivity and specificity.

Main Methods:

• ssROC utilizes a small labeled dataset to nonparametrically impute missing labels.
• Imputed labels are then used for ROC parameter estimation.
• The performance of ssROC was evaluated using synthetic, semi-synthetic, and real-world EHR data.

Main Results:

• ssROC demonstrated minimal bias and significantly lower variance in ROC parameter estimates compared to classical supervised ROC analysis (supROC).
• In simulations, ssROC outperformed supROC in precision.
• For 5 PAs from Mass General Brigham (MGB) EHR data, ssROC estimates were 30% to 60% less variable than supROC on average.

Conclusions:

• ssROC enables precise evaluation of PA performance without requiring large volumes of labeled data.
• The ssROC method is easily implementable using open-source R software.
• Integrating ssROC with weakly-supervised PAs facilitates reliable and streamlined EHR-based phenotyping for research.