Adaptive Multi-Wave Sampling for Efficient Chart Validation

To use computable phenotypes for characterization of patients and identification of outcomes in studies conducted using healthcare claims and electronic health record data. Chart review studies establish reference labels against which computable phenotypes are compared to understand their measurement characteristics (the quantity of interest), for instance, the positive predictive value.

We describe a method to adaptively evaluate a quantity of interest over sequential samples of charts, with the goal to minimize the number of charts reviewed. With the help of a simultaneous confidence band, we stop the reviewing once the confidence band meets a pre-specified stopping threshold. The contribution of this work is threefold. First, we test the use of an adaptive approach called Neyman's sampling versus random or stratified random sampling. Second, we propose frequentist confidence bands and Bayesian credible intervals to sequentially evaluate the quantity of interest. Third, we propose a tool to predict the stopping time (defined as the number of charts reviewed) at which the chart review would be complete.

We observe that Bayesian credible intervals prove to be tighter than its frequentist confidence band counterparts. Moreover, we observe that simple random sampling often performs similarly to Neyman's sampling.

We conclude that random sampling in connection with Bayesian credible intervals allows for an efficient validation of a response of interest for both binary and continuous outcomes.