Incorporating Auxiliary Information into Assessment of Accuracy and Discrimination of Risk Models When Some...

Validating a risk model in independent data is an important step before recommending it for broad use. However, some model predictors may be missing for some subjects in the validation data, either randomly or by design. Missingness by design occurs for case-cohort or nested case-control studies, in which some predictors are measured only for sub-sampled subjects. Standard approaches to handling missing data are weighting and imputation. We propose adjusting known sampling weights by incorporating auxiliary information available for all cohort members to improve the efficiency of weighted estimates of measures of classification accuracy and discrimination for a risk prediction model when validating it in sub-samples of a cohort. We study estimates of the true and false positive rates for specific risk thresholds, the area under the receiver operator characteristic curve (AUC), the proportion of cases followed (to assess the usefulness of models for screening applications), and the positive and negative predictive values. We use influence functions as auxiliary variables for efficient weight adjustment and compare the efficiency of the resulting estimates to that based on weights adjusted using more heuristically derived auxiliary variables. We derive analytic variance estimates that incorporate the weight estimation. We compare using weight adjustment to using multiple imputation in simulations. While multiple imputation was often efficient, it yielded biased estimates of the performance measures for mis-specified imputation models. To illustrate these methods further we assess the performance of an absolute risk model for second primary thyroid cancer in an independent cohort.