Inter-observer variability among pathologists' evaluation of malignant melanoma: effects upon an analytic study

This study examined whether inter-observer variability in rating tumor characteristics affected results of an investigation of surveillance bias and malignant melanoma at the Lawrence Livermore National Laboratory. The 20 cases from the Laboratory and their 36 non-Laboratory controls belonged to the same pre-paid health plan and were diagnosed with melanoma between 1970 and 1984. Tumors were independently and then jointly rated by three dermatopathologists blind to the subjects' Laboratory status. The mean difference between the reviewers and the consensus reading for tumor thickness was small, ranging from -0.06 mm (95% confidence interval [CI]--0.12, 0.00) to 0.00 mm (95% CI--0.07, 0.07). Agreement was much lower for histologic type (kappa = 0.48, 95% CI 0.37, 0.58). Because the inter-observer variability, the study's hypothesis was rejected by analyses based on data from the consensus reading and two reviewers, but not on data from the third reviewer. These findings suggest that epidemiologists using data subject to inter-observer variability may want to employ consensus instead of individual ratings.