Measuring the reliability of clinical data: the case for using three observers

Misclassification of clinical data occurs commonly, because of measurement or judgmental errors, or because of bias in the observational method. The sensitivity and specificity of a given observer (or method) may sometimes be estimated by comparison with an independent "definitive" diagnosis; this is not routinely practical, however, if the "definitive" method involves additional risk or cost. A second possibility is to compare two independent, fallible observers, using an index of agreement such as the kappa statistic; the misclassification probabilities are not then estimable, and the agreement indices have the disadvantage of being dependent on the population prevalence. In contrast, if three or more independent observers are used, their reliability may be assessed by the direct estimation of sensitivity and specificity. An example is given of a series of patients who were independently evaluated by three radiologists.