Estimating the decision curve and its precision from three study designs

The decision curve plots the net benefit $\left(NB\right)$ of a risk model for making decisions over a range of risk thresholds, corresponding to different ratios of misclassification costs. We discuss three methods to estimate the decision curve, together with corresponding methods of inference and methods to compare two risk models at a given risk threshold. One method uses risks (R) and a binary event indicator (Y) on the entire validation cohort. This method makes no assumptions on how well-calibrated the risk model is nor on the incidence of disease in the population and is comparatively robust to model miscalibration. If one assumes that the model is well-calibrated, one can compute a much more precise estimate of $NB$ based on risks R alone. However, if the risk model is miscalibrated, serious bias can result. Case-control data can also be used to estimate $NB$ if the incidence (or prevalence) of the event ( $Y=1$ ) is known. This strategy has comparable efficiency to using the full $(R,Y)$ data, and its efficiency is only modestly less than that for the full $(R,Y)$ data if the incidence is estimated from the mean of Y. We estimate variances using influence functions and propose a bootstrap procedure to obtain simultaneous confidence bands around the decision curve for a range of thresholds. The influence function approach to estimate variances can also be applied to cohorts derived from complex survey samples instead of simple random samples.