Use of Sparse Internal Validation Data

Internal validation sub-studies compare an observed imperfect measurement of a variable with its gold standard measurement in a sample of participants. The gold standard is often expensive or difficult to collect, so the validation sample is usually small. We simulated a small cohort study with a nested, balanced validation sub-study and used the simulated cohorts to evaluate three analytic strategies with the objective of determining which of them, alone or in combination, improved the performance of quantitative bias adjustments informed by the validation sub-study. The strategies were: (1) incorporating a weak prior to the beta distributions assigned to the bias parameters, as opposed to no prior (2) restricting beta distribution assignment to the three predictive values informed by sampling, as opposed to all four predictive values, and (3) restricting imputation of gold-standard measurement to study cohort members who were not sampled for the validation sub-study, as opposed to imputing for all members of the study cohort. All twelve combinations of bias adjustment strategies returned bias-adjusted estimates with less bias and higher coverage than the simulated misclassified cohorts, although all had higher root mean squared errors and wider intervals because of the uncertainty added by incorporating the sparse validation data into the analysis. We found that assigning a prior may sometimes have utility, whereas restricting assignment of beta distributions for use only with sampling and restricting imputation to only cohort members not selected for the validation study adds computational complexity with little utility in return.