EVALUATING RISK-PREDICTION MODELS USING DATA FROM ELECTRONIC HEALTH RECORDS.

Area of Science:

• Health Informatics
• Biostatistics
• Clinical Epidemiology

Background:

• Electronic health records (EHRs) enable risk-prediction model development.
• Outcome misclassification, due to uncaptured events (false negatives), can limit accuracy estimation.
• Robustness of prediction accuracy summaries needs evaluation under misclassification.

Purpose of the Study:

• To evaluate the robustness of prediction accuracy summaries (e.g., receiver operating characteristic curves, risk-reclassification) when outcomes are misclassified.
• To quantify potential bias in prediction accuracy summaries if misclassification depends on marker values.
• To compare prognostic models for 30-day hospital readmission in heart failure patients.

Main Methods:

• Derivation of estimators for sensitivity and specificity assuming independent misclassification.
• Simulation studies to quantify bias in prediction accuracy summaries when misclassification depends on marker values.
• Comparison of alternative prognostic models using real-world data for heart failure readmissions.

Main Results:

• Simulation studies showed that misclassification dependent on marker values biases estimated accuracy improvement.
• The direction of bias in accuracy estimation depends on the association between markers and misclassification probability.
• In application, 29% of readmissions were to external hospitals, impacting prediction accuracy.

Conclusions:

• Outcome misclassification can lead to erroneous conclusions about the accuracy of risk-prediction models.
• Risk-prediction model accuracy is sensitive to how accurately outcomes are captured in EHRs.
• Careful consideration of potential misclassification is crucial for reliable risk prediction.