Comparative Large Language Models Evaluation for Age-Related Macular Degeneration Diagnosis on OCT and Rubric-Based...

To develop and validate a rubric-based large language model (LLM) evaluator for assessing the diagnostic quality of LLM-generated age-related macular degeneration (AMD) diagnoses from OCT scans, benchmarked against human evaluators, and, as a secondary aim, to characterize the diagnostic capability of the underlying LLMs.

Retrospective study primarily developing and validating a rubric-based LLM evaluator for assessing the diagnostic quality of LLM-generated diagnoses on retinal OCT, with the diagnostic performance of the LLMs assessed as a secondary analysis.

A total of 140 OCT scans (70 of choroidal neovascular membrane, 70 of drusen), 6 diagnoser models (GPT-4.1, GPT-4o, o4-mini, Gemini-2.5-Flash, GPT-5-mini, and GPT-5.5), and 4 evaluator models (GPT-4.1, GPT-5-mini, o4-mini, and Gemini-2.5-Flash) were included in this study.

We developed a 2-stage workflow to automate OCT diagnosis associated with AMD. First, LLM "diagnosers" are prompted to provide a diagnosis in natural language from OCT scans. Second, LLM "evaluators" grade the diagnosis against a predetermined rubric for accuracy, imaging features, differential diagnosis, and clarity. A pilot experiment on 20 OCT scans benchmarked four LLM evaluators against a human evaluator using the identical rubric. Interrater reliability between the selected LLM evaluator and 2 retinal specialists was further assessed across 560 image-diagnosis pairs using intraclass correlation coefficients (ICCs) and two-1-sided test equivalence analysis. The best-performing LLM evaluator was then deployed in a full-scale analysis on 140 OCT scans to grade all 6 diagnoser models.

Average gradings, standard deviation, P value, correlation coefficient, ICC values, paired t-test P values, and two-1-sided test equivalence bounds between human and LLM evaluators.

In the pilot analysis, Gemini-2.5-Flash demonstrated the strongest alignment with the human evaluator. In the expanded experiment across 560 image-diagnosis pairs, ICC analysis confirmed that Gemini-2.5-Flash's agreement with human specialists (ICC = 0.897) was comparable to interspecialist agreement (ICC = 0.806), and two-1-sided test analysis supported statistical equivalence with human scoring, with a mean difference of 0.123 points. In the large-scale evaluation, GPT-5.5 achieved the highest average diagnoser score (6.3/8), followed by GPT-5-mini (5.0) and Gemini-2.5-Flash (4.9).

Large language model evaluators demonstrate a high level of agreement with human evaluator in supervising the quality of AMD diagnoses on OCT scans generated by LLM diagnosers. This workflow demonstrates the potential of LLM-powered automation for diagnostic quality evaluation in ophthalmology clinical practice.

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.