Can OpenAI's New o1 Model Outperform Its Predecessors in Common Eye Care Queries?

Area of Science:

• Ophthalmology
• Artificial Intelligence in Healthcare
• Medical Informatics

Background:

• The rapid advancement of AI models like OpenAI's o1 promises enhanced reasoning for specialized queries.
• Assessing the performance of new AI models in critical fields such as eye care is essential.
• Previous AI models have shown variable performance in responding to ophthalmic questions.

Purpose of the Study:

• To evaluate and compare the performance of OpenAI's o1, ChatGPT-4o, and ChatGPT-4 in answering ophthalmic-related queries.
• To assess AI models based on correctness, completeness, and readability in the context of eye care.
• To identify strengths and weaknesses of o1 in addressing specific ophthalmic subtopics.

Main Methods:

• A cross-sectional study design was employed.
• Sixteen challenging ophthalmic queries (myopia, ocular symptoms, retinal conditions) were used.
• Responses were evaluated by three masked ophthalmologists on correctness, completeness, and readability (5-point scale).

Main Results:

• OpenAI's o1 achieved the highest scores for correctness (12.6) and readability (14.2), outperforming ChatGPT-4.
• No significant performance difference was observed between o1 and ChatGPT-4o.
• ChatGPT-4o led in completeness (12.4), with o1 scoring 10.8; o1 showed limitations in completeness for ocular symptom queries (5.5/15).

Conclusions:

• OpenAI's o1 demonstrates superior correctness and readability over ChatGPT-4 for ophthalmic queries, aligning with ChatGPT-4o's performance.
• Despite marketing claims, o1's overall performance in eye care queries is comparable to ChatGPT-4o.
• Further improvements in completeness, particularly for specific conditions like ocular symptoms, are needed for o1.