Assessing surveillance using sensitivity, specificity and timeliness

Monitoring ongoing processes of illness to detect sudden changes is an important aspect of practical epidemiology and medicine more generally. Most commonly, the monitoring has been restricted to a unidimensional stream of data over time. In such situations, analytic results from the industrial process monitoring have suggested optimal approaches to monitor the data streams. Data streams including spatial location as well as temporal sequence are becoming available. Monitoring methods that incorporate spatial data may prove superior to those that ignore it. However, analytically, optimal methods for spatial surveillance data may not exist. In the present article, we introduce and discuss evaluation metrics that can be used to compare the performance of statistical methods of surveillance. Our general approach is to generalize receiver operating characteristic (ROC) curves to incorporate the time of detection in addition to the usual test characteristics of sensitivity and specificity. In addition to weighting ordinary ROC curves by two measures of timeliness, we describe three three-dimensional generalizations of ROC curves that result in timeliness-ROC surfaces. Working in the context of surveillance of cases of disease to detect a sudden outbreak, we demonstrate these in an artificial example and in a previously described simulation context and show how the metrics differ. We also discuss the differences and under which circumstances one might prefer a given method.