DeepSeek-R1 vs OpenAI o1 for Ophthalmic Diagnoses and Management Plans

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Clinical Decision Support

Background:

• Large language models (LLMs) are increasingly explored for clinical decision-making.
• Few studies have assessed LLM performance on complex ophthalmology cases from clinical practice.
• Evaluating open-weight vs. proprietary LLMs is crucial for clinical utility and accessibility.

Purpose of the Study:

• To compare the diagnostic accuracy and management decision-making of DeepSeek-R1 and OpenAI o1.
• To assess the cost-effectiveness of these LLMs in ophthalmology.
• To understand the performance of reasoning-enhanced, open-weight LLMs in complex clinical scenarios.

Main Methods:

• Cross-sectional evaluation of DeepSeek-R1 and OpenAI o1 using standardized prompts (Plan-and-Solve Plus).
• Utilized 422 complex clinical cases from JAMA Ophthalmology's Clinical Challenge articles across 10 subspecialties.
• Assessed diagnostic accuracy, next-step decision-making accuracy, and token costs.

Main Results:

• DeepSeek-R1 demonstrated higher diagnostic accuracy (70.4%) than OpenAI o1 (63.0%) (P=.02).
• DeepSeek-R1 also showed superior next-step decision accuracy (82.7%) compared to OpenAI o1 (75.8%) (P=.01).
• DeepSeek-R1 offered significantly lower costs, with savings up to 98.5%.

Conclusions:

• DeepSeek-R1 outperformed OpenAI o1 in diagnostic and management accuracy for ophthalmology cases.
• Open-weight LLMs, like DeepSeek-R1, present a cost-effective and scalable solution for clinical decision support.
• Further research is needed on safety and domain-specific adaptations for optimal clinical integration.