Related Experiment Video
Updated: Sep 14, 2025
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
OpenAI o1 Large Language Model Outperforms GPT-4o, Gemini 1.5 Flash, and Human Test Takers on Ophthalmology
Ryan Shean1, Tathya Shah1, Sina Sobhani1
1Keck School of Medicine, University of Southern California, Los Angeles, California.
Purpose:
To evaluate and compare the performance of human test takers and three artificial intelligence (AI) models-OpenAI o1, ChatGPT-4o, and Gemini 1.5 Flash-on ophthalmology board-style questions, focusing on overall accuracy and performance stratified by ophthalmic subspecialty and cognitive complexity level.
Design:
A cross-sectional study.
Subjects:
Five hundred questions sourced from the Basic and Clinical Science Course (BCSC) and EyeQuiz question banks.
Methods:
Three large language models interpreted the questions using standardized prompting procedures. Subanalysis was performed, stratifying the questions by subspecialty and complexity defined by the Buckwalter taxonomic schema. Statistical analysis, including the analysis of variance and McNemar test, was conducted to assess performance differences.
Main Outcome Measures:
Accuracy of responses for each model and human test takers, stratified by subspecialty and cognitive complexity.
Results:
OpenAI o1 achieved the highest overall accuracy (423/500, 84.6%), significantly outperforming GPT-4o (331/500, 66.2%; P < 0.001) and Gemini (301/500, 60.2%; P < 0.001). o1 demonstrated superior performance on both BCSC (228/250, 91.2%) and EyeQuiz (195/250, 78.0%) questions compared with GPT-4o (BCSC: 183/250, 73.2%; EyeQuiz: 148/250, 59.2%) and Gemini (BCSC: 163/250, 65.2%; EyeQuiz: 137/250, 54.8%). On BCSC questions, human performance was lower (64.5%) than Gemini 1.5 Flash (65.2%), GPT-4o (73.2%), and OpenAI o1 (91.2%) (P < 0.001). OpenAI o1 outperformed other models in each of the nine ophthalmic subfields and three cognitive complexity levels.
Conclusions:
OpenAI o1 outperformed GPT-4o, Gemini, and human test takers in answering ophthalmology board-style questions from two question banks and across three complexity levels. These findings highlight advances in AI technology and OpenAI o1's growing potential as an adjunct in ophthalmic education and care.
Financial Disclosures:
The author(s) have no proprietary or commercial interest in any materials discussed in this article.
More Related Videos
04:48Application of Deep Learning-Based Medical Image Segmentation via Orbital Computed Tomography
Published on: November 30, 2022
13:12Translational Brain Mapping at the University of Rochester Medical Center: Preserving the Mind Through Personalized Brain Mapping
Published on: August 12, 2019
