Testing calibration of phenotyping models using positive-only electronic health record data

Area of Science:

• Biomedical Informatics
• Clinical Data Science
• Health Services Research

Background:

• Validating phenotyping models with Electronic Health Records (EHRs) typically requires labor-intensive gold-standard case and control labels.
• Identifying gold-standard controls is challenging for certain diseases, making model validation difficult.
• Existing EHR data often comprises labeled cases and numerous unlabeled patients, a common scenario for model development.

Purpose of the Study:

• To propose novel methods for assessing model calibration and discrimination using
• positive-only
• EHR data.
• To enable model validation without requiring gold-standard controls, provided labeled cases are representative.

Main Methods:

• Developed a novel statistic for model calibration assessment based on differences between model-free and model-based case estimations across risk subgroups.
• Proposed consistent estimators for discrimination measures and derived their large sample properties.
• Demonstrated the utility of these methods for estimating calibration slope using positive-only data.

Main Results:

• The proposed calibration statistic asymptotically follows a Chi-squared distribution.
• Calibration slope estimation is feasible using positive-only EHR data.
• Consistent estimators for discrimination measures were developed with derived large sample properties.
• Methods were validated through extensive simulations and applied to EHR data for primary aldosteronism risk prediction.

Conclusions:

• The proposed methods offer a viable approach to validate phenotyping models using EHR data lacking gold-standard controls.
• These techniques significantly reduce the reliance on time-consuming manual chart reviews by clinical experts.
• The findings facilitate more efficient and scalable model development and validation in clinical informatics.