The effect of verification on the assessment of imaging techniques

In order to measure the accuracy of a diagnostic imaging technique, the correctness of each observer decision must be determined. This is often difficult in clinical cases, and various strategies are frequently used to approximate the truth. Experiments using chest radiographs as an example are reported to show the pitfalls in those strategies. To assess relative accuracies of three different radiographic techniques, chest radiographs were taken of 66 patients with each of the three techniques, and the films were evaluated by six radiologists. Their findings were then scored by comparing them with the correct decision defined by each of the following methods: majority vote, consensus opinion, expert judgment, feedback review, and clinical/pathologic proof. The findings showed that any one of the techniques could be shown to be better than the other, depending on how the truth was defined. It is concluded, therefore, that strategies that define the presence or absence of disease only by the diagnostic tests under evaluation are inadequate. The truth must be determined by clinical or pathologic proof and follow-up data.