Corrected ROC analysis for misclassified binary outcomes.

Area of Science:

• Biostatistics
• Health Informatics
• Machine Learning in Healthcare

Background:

• Electronic Health Records (EHRs) are vital for precision medicine but contain imperfect data.
• Outcome misclassification in EHR data can significantly bias risk prediction models and accuracy metrics.
• Standard Receiver Operating Characteristic (ROC) analysis, particularly the Area Under the Curve (AUC), is susceptible to misclassification bias.

Purpose of the Study:

• To investigate the impact of outcome misclassification on the accuracy assessment of risk prediction models using EHR data.
• To develop and introduce a novel misclassification-adjusted ROC procedure for bias-corrected AUC estimation.
• To demonstrate the effectiveness of the proposed method on a large-scale EHR dataset.

Main Methods:

• Studied the effect of misclassification on AUC bias in regression prediction models.
• Introduced an intuitive misclassification-adjusted ROC procedure to account for outcome uncertainty.
• Applied the correction method to a hospitalization prediction model using EHR data from over 1 million patients.

Main Results:

• Misclassification of outcomes in EHR data leads to biased AUC estimates in standard ROC analysis.
• Bias in AUC is influenced by false positive/negative rates and disease prevalence.
• The proposed misclassification-adjusted ROC procedure effectively produces bias-corrected AUC estimates, outperforming simple correction during model building.

Conclusions:

• Accurate risk prediction from EHRs requires addressing outcome misclassification.
• The novel misclassification-adjusted ROC procedure offers a computationally simple and effective way to correct AUC bias.
• This method is crucial for reliable model comparison and development in precision medicine using large EHR datasets.