Variability in grading diabetic retinopathy from stereo fundus photographs: comparison of physician and lay readers

Two physicians and two lay readers were trained according to a detailed protocol in the grading of 17 lesions found in diabetic retinopathy by evaluation of stereo fundus photographs according to a modified Airlie House classification. Intraobserver and interobserver variability of these readers was assessed by two methods: weighted kappa, and frequency of agreement within one grade. In general, physician readers were found to be less variable on replicate readings than were lay leaders, and had slightly better agreement with each other than with the lay readers. The physiological significance of the direction and magnitude of the difference between physician and lay reader variability for individual lesions was often uncertain. Assessment of contribution to disagreement by individual readers was possible and permits future training directed at reducing disagreement to acceptable values.