Reliability assessment of a rod photoreceptor outer segment grading system

The purpose of this study was to assess the reliability of a rod photoreceptor outer segment (PR-OS) grading system based on the analysis of 1 microm thick retinal sections obtained from Xenopus laevis whole-eye organ cultures. Digitally captured images, representative of the entire spectrum of rod PR-OS organization levels, were selected and coded numerically. A total of 102 individual rod PR-OS profiles were graded according to a six-step classification scheme based on the percentage of rod PR-OS membrane organization. Unweighted (exact agreement) and weighted kappa (kappa) coefficients (for use with ordered categorical rating scales) were calculated. Differences between kappa coefficients were tested for by chi-square analysis. To investigate the intra- and inter-rater variability and the possible presence of an interaction of the measurements with time, a repeated-measures analysis of variance was performed. The overall unweighted and weighted intra-rater kappa coefficients were 0.78 and 0.92, respectively. The overall unweighted and weighted inter-rater kappa coefficients were 0.73 and 0.90, respectively. There was no significant difference between raters or between first and second reading, nor was interaction between raters and time of rating documented. Individual kappa coefficients were equivalent both between raters and between sessions. Intra- and inter-rater agreement was within one step in 100% of cases. The estimated values of the kappa coefficients are consistent with a good to excellent degree of reliability and reproducibility of this rod PR-OS grading system. This system will be useful in the assessment of rod PR-OS morphology in studies of photoreceptor physiology and pathology.