Assessing intergrader variability in incomplete outer retinal atrophy (iRORA) grading

To assess the intergrader variability in detecting incomplete outer retinal atrophy (iRORA), an optical coherence tomography (OCT) biomarker for early atrophic changes in age-related macular degeneration (AMD) and a potential clinical trial endpoint.

We performed a cross-sectional intergrader agreement study using 60 OCT B-scans enriched for iRORA features from non-exudative AMD eyes. Five international retinal experts independently graded images for iRORA presence according to the current Classification of Atrophy Meeting (CAM) criteria, using a standardised online annotation platform. Pairwise agreement was assessed with Cohen's Kappa; agreement with the majority vote, sensitivity and specificity were calculated using a leave-one-out approach. Annotation lead times were recorded to explore links between decision speed and grading consistency.

Mean pairwise Cohen's Kappa was 0.425 (range 0.19-0.73), indicating mild agreement. Agreement with the majority vote ranged from κ=0.48 to 0.67, with classification accuracy between 0.73 and 0.83. Sensitivity varied from 0.59 to 0.83, while specificity remained high (0.85-0.96). Annotation lead times were generally shorter when grader decisions aligned with the majority consensus.

Despite standardised CAM definitions, intergrader reproducibility for iRORA detection remains limited, reflecting challenges in consistent early atrophy identification. Refinement of grading guidelines and integration of automated detection systems could improve reliability for AMD clinical trials and clinical practice.