Prognostic scores for detecting a high risk group: estimating the sensitivity when applied to new data

The sensitivity of a prognostic scoring system will tend to be exaggerated if the scoring system is both derived and validated on the same data. This paper provides, by analogy to regression with error in an explanatory variable, an intuitive basis for the methodological results of Copas which seek to estimate the degree of such exaggeration. There was good agreement between Copas' results and those achieved in a series of cross-validation exercises where logistic regression models predicting the risk of ischaemic heart disease were derived using data from the prospective British Regional Heart Study. When truly important variables were included, the exaggeration of the sensitivity increased as the number of cases of disease available decreased. It is concluded that Copas' method, which is easy to implement in practice, may be helpful in realistically anticipating the extent of such exaggeration, and that it can be usefully employed before pursuing a scoring system on newly collected data.