ChatGPT-4o and OpenAI-o1: A Comparative Analysis of Its Accuracy in Refractive Surgery.

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Education

Background:

• Assessing AI accuracy in medical knowledge retrieval.
• Evaluating AI performance against human experts in specialized fields.
• Investigating AI's role in clinical decision support.

Purpose of the Study:

• To compare the accuracy of ChatGPT-4o and OpenAI-o1 in answering refractive surgery questions.
• To evaluate if AI performance can support clinical decision-making.
• To benchmark AI against ophthalmology residents and clinicians.

Main Methods:

• Randomized, questionnaire-based study.
• 228 text-only questions from AAO BCSC Self-Assessment Program (Refractive Surgery).
• Comparison of AI models (ChatGPT-4o, OpenAI-o1) against 1983 human users.

Main Results:

• OpenAI-o1 scored 91.2%, ChatGPT-4o scored 86.4%.
• Both AI models significantly outperformed human users (average 77%).
• AI performance difference was not statistically significant (p=0.1045).

Conclusions:

• ChatGPT-4o and OpenAI-o1 demonstrate high accuracy in refractive surgery knowledge.
• AI models significantly outperform human experts in this domain.
• AI's potential to augment medical decision-making requires further investigation and clinical outcome studies.