Postprediction Inference for Clinical Characteristics Extracted With Machine Learning on Electronic Health Records.

Area of Science:

• Biostatistics
• Health Informatics
• Machine Learning in Healthcare

Background:

• Real-world data (RWD) from electronic health records (EHRs) are crucial for understanding cancer outcomes.
• Machine learning (ML) offers a scalable method for extracting patient characteristics from unstructured clinical notes.
• Standard ML metrics do not fully capture the impact of extracted data error on downstream analytical results.

Purpose of the Study:

• To define and evaluate "postprediction inference" for statistical models using ML-extracted variables.
• To assess methods for recovering valid estimation and inference from ML-derived covariates.
• To investigate the utility of labeled data in improving inference from ML-extracted variables.

Main Methods:

• Developed four postprediction inference approaches for Cox proportional hazards models with binary ML-extracted covariates.
• Evaluated methods using ML-predicted probabilities and, for some, a labeled validation dataset.
• Tested approaches on simulated data and EHR-derived RWD from a national cancer cohort.

Main Results:

• Postprediction inference methods can improve the accuracy of statistical modeling using ML-extracted variables.
• High-performing ML models generally yield valid estimation and inference.
• Incorporating auxiliary labeled data further enhances the reliability of results.

Conclusions:

• Methods for fitting statistical models with ML-extracted variables subject to error are presented and evaluated.
• Leveraging even limited labeled data can significantly improve inference from ML-extracted variables.
• Postprediction inference provides a framework for more robust use of ML in health research.