Ophthalmological Question Answering and Reasoning Using OpenAI o1 vs Other Large Language Models.

Area of Science:

• Artificial Intelligence
• Ophthalmology
• Medical Informatics

Background:

• Large Language Models (LLMs) are increasingly evaluated for specialized medical applications.
• OpenAI's o1 LLM, with dedicated reasoning capabilities, requires assessment in ophthalmology.
• General LLM reasoning may not suffice for specialized medical domains, necessitating domain-specific models.

Purpose of the Study:

• To evaluate the performance and reasoning abilities of OpenAI's o1 LLM against other leading LLMs using ophthalmology-specific questions.
• To determine if o1's general reasoning capabilities meet the demands of specialized medical fields.
• To inform the development and necessity of domain-specific LLMs in ophthalmology.

Main Methods:

• Six LLMs (o1, GPT-4o, GPT-4, GPT-3.5, Llama 3-8B, Gemini 1.5 Pro) were tested on 6990 ophthalmology questions from the MedMCQA dataset.
• Performance was measured by accuracy and macro F1 score.
• Reasoning abilities were assessed using text-generation metrics (ROUGE-L, BERTScore, BARTScore, AlignScore, METEOR) and expert qualitative evaluation for usefulness and organization.

Main Results:

• LLM o1 achieved the highest accuracy (0.877) and macro F1 score (0.877).
• GPT-4o and GPT-4 outperformed o1 in BERTScore and AlignScore metrics.
• o1 demonstrated superior performance in BARTScore and METEOR, and expert reviews rated its responses as more useful and organized than GPT-4o.

Conclusions:

• OpenAI's o1 LLM exhibits strong performance in accuracy for ophthalmology questions but shows variability in text-generation quality.
• While o1 offers promising clinical utility and organization, its performance suggests that domain-specialized LLMs may still be required for optimal ophthalmology applications.
• Further targeted evaluations are recommended to fully understand LLM capabilities in specialized medical fields.