Performance of GPT-5 Frontier Models in Ophthalmology Question Answering.

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Ophthalmology

Background:

• Large language models (LLMs) show promise for complex medical question-answering.
• Optimizing accuracy and cost-efficiency for advanced LLMs like Generative Pretrained Transformer-5 (GPT-5) is crucial.
• Previous LLMs require evaluation for specialized medical domains.

Purpose of the Study:

• To evaluate the performance and cost-accuracy trade-offs of OpenAI's GPT-5.
• To compare GPT-5 against previous generation LLMs on ophthalmic question answering.
• To establish optimal configurations for accuracy and cost-efficiency in medical AI.

Main Methods:

• Evaluated 12 GPT-5 configurations against o1-high, o3-high, and GPT-4o.
• Utilized 260 multiple-choice questions from the American Academy of Ophthalmology dataset.
• Assessed accuracy, ranking via Bradley-Terry model, and rationale quality using an LLM-as-a-judge framework.

Main Results:

• GPT-5-high achieved the highest accuracy (0.965), significantly outperforming GPT-4o and other variants.
• GPT-5-high ranked first in accuracy and rationale quality compared to o3-high.
• Cost-accuracy analysis identified GPT-5-mini-low as an optimal low-cost, high-performance configuration.

Conclusions:

• GPT-5 with high reasoning effort achieves near-perfect accuracy in ophthalmology question answering.
• GPT-5 surpasses prior reasoning LLMs, establishing a new benchmark for medical AI.
• An automated LLM-as-a-judge framework enables scalable evaluation of medical AI responses.