Evolving Consultation: Enhancing Ophthalmic Diagnostic Performance Using Large Language Model

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Education

Background:

• Large language models (LLMs) are increasingly used in healthcare.
• Studies assessing LLM effectiveness in ophthalmology for complex differential diagnoses are lacking.
• This study evaluates ChatGPT-4o's impact on ophthalmologists' clinical reasoning.

Purpose of the Study:

• To assess the effectiveness of ChatGPT-4o in improving ophthalmologists' diagnostic reasoning.
• To determine which experience levels benefit most from LLM assistance.
• To analyze changes in response quality, factuality, and safety.

Main Methods:

• Prospective study involving 20 ophthalmologists (10 residents, 10 board-certified).
• Ten original ophthalmic clinical scenarios were used.
• Responses were collected before and after ChatGPT-4o assistance and evaluated on coherency, factuality, comprehensiveness, and safety.

Main Results:

• ChatGPT-4o significantly improved coherency, comprehensiveness, and safety scores for both groups (P < 0.001).
• Factuality scores did not significantly improve (P = 0.114 and 0.839).
• Increased citation frequency was observed, but 44% were inaccurate; variability in factuality and safety increased.

Conclusions:

• ChatGPT-4o enhances diagnostic reasoning and response quality, particularly for residents.
• Integration requires managing increased variability in factuality and safety.
• Retrieval-augmented generation systems may ensure accurate and safe AI-assisted clinical information.