Measuring inter-observer agreement in contour delineation of medical imaging in a dummy run using Fleiss' kappa

In medical imaging used for planning of radiation therapy, observers delineate contours of a treatment volume in a series of images of uniform slice thickness.

To summarize agreement in contouring between an arbitrary number of observers by a single number, we generalized the kappa index proposed by Zijdenbos et al. (1994).

Observers characterized voxels by allocating them to one of two categories, inside or outside the contoured region. Fleiss' kappa was used to measure association between n indistinguishable observers. Given the number Vi of voxels contoured by exactly i observers (i = 1, ..., n ), the resulting overall kappa is representable as a ratio of weighted sums of the Vi .

Overall kappa was applied to analyze inter-center variations in a multicenter trial on radiotherapy planning in patients with locally advanced lung cancer. A contouring dummy run was performed within the quality assurance program. Contouring was done twice, once before and once after a training program. Observer agreement was enhanced from 0.59 (with a 95% confidence interval (CI) of 0.51-0.67) to 0.69 (95% CI 0.59-0.78).

By contrast to average pairwise indices, overall kappa measures observer agreement for more than two observers using the full information about overlapping volumes, while not distinguishing between observers. It is particularly adequate for measuring observer agreement when identification of observers is not possible or desirable and when there is no gold standard.