Related Experiment Video
Updated: Oct 27, 2026
Comparison of Agreement and Accuracy using Binocular Wavefront Optometer with Autorefractor and Phoropter
Published on: September 16, 2025
OpenEvidence versus Six Leading Large Language Models in Refractive Surgery: Does Curated Medical Evidence Confer an
Avi Wallerstein1,2,3, Mathilde Gallant3,4, Mathieu Gauvin1,3
1Department of Ophthalmology and Visual Sciences, McGill University, Montreal, QC, Canada.
Purpose:
To assess the accuracy of OpenEvidence relative to six leading large language models (LLMs) in answering refractive surgery multiple-choice questions, and to compare their performance with users of the American Academy of Ophthalmology (AAO) Basic and Clinical Science Course (BCSC) Self-Assessment Program.
Setting:
Not applicable.
Design:
Randomized questionnaire-based study using 225 questions from the refractive surgery section of the AAO BCSC Program.
Methods:
OpenEvidence and 6 LLMs (ChatGPT-4o, ChatGPT-5, Claude, Copilot, Gemini, and Grok) answered the same set of multiple-choice questions. Their accuracy was defined as the proportion of correct responses. Performance from 1,983 BCSC Self-Assessment Program users served as a benchmark.
Results:
OpenEvidence achieved the lowest accuracy (76%) and performed significantly worse (P<0.05) than all six general-purpose LLMs. Claude achieved the highest accuracy (92%), followed by Grok (89.8%). ChatGPT-5, Copilot, and Gemini each scored an accuracy of 88%, and ChatGPT-4o achieved 86.2%. All models' lower 95% confidence interval bounds exceeded the 60% passing threshold. The average score of BCSC Self-Assessment Program users was 77%. All general-purpose LLMs scored significantly higher than BCSC Self-Assessment Program users (P<0.05), whereas OpenEvidence did not.
Conclusions:
OpenEvidence demonstrated significantly lower accuracy than the six evaluated general-purpose LLMs and did not exceed the BCSC Self-Assessment program user benchmark on this standardized refractive surgery multiple-choice assessment. This task-specific difference was not fully explained by response-format noncompliance and should not be interpreted as direct evidence of globally inferior medical knowledge or performance in open-ended clinical applications.