Observer agreement and accuracy for lung scintigraphy: a comparison with pulmonary angiography

This study investigated whether the moderate observer agreement rate for lung scintigraphy, when using the PIOPED criteria, reflects significant variations in accuracy. In a prospective study, 170 patients were investigated with lung scintigraphy and pulmonary angiography. The lung scintigrams were read according to the PIOPED criteria by three skilled observers from different hospitals who had no experience of consensus interpretation. The scintigrams were reinterpreted 6 months later. The inter-observer agreement rates were moderate and fair, with kappa values of 0.49 (0.44-0.54) for the first reading and 0.36 (0.31-0.41) for the second reading. The intra-observer agreement rates were good and moderate, with kappa values of 0.62 (0.51-0.73), 0.52 (0.42-0.62) and 0.54 (0.44-0.64). There were no significant differences between the observers in the areas under the ROC curves. However, one observer showed a significant difference between his first and second reading (P = 0.02). All three observers showed good accuracy. Although there were substantial variations in the inter- and intra-observer agreement rates, this was only reflected to a minor degree in the accuracy.