An application of kappa-type analyses to interobserver variation in classifying chest radiographs for pneumoconiosis

We investigated interobserver variation between three physician readers in the independent classification of chest radiographs from 1771 taconite workers for pneumoconiosis. We analysed variation with a general methodology for the analysis of categorical data, and quantified interobserver agreement in terms of kappa-type statistics. The results revealed considerable variation in the profusion of opacities reported by each observer. This was especially so for the earliest radiographic changes, but variation also occurred with the small number of films in the series showing category 2 pneumoconiosis or over. Variation in the classification of profusion of small opacities was greater in films of poor quality, and also increased with the length of time between taking and reading the film (film age). We could not account for the effect of film age by controlling for film quality, as subjectively assessed by the readers.