Enhancing diabetic retinopathy query responses: assessing large language model in ophthalmology

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Informatics

Background:

• Diabetic retinopathy (DR) is a major cause of blindness.
• Large language models (LLMs) are increasingly used for health information.
• The accuracy of LLM responses for DR queries requires evaluation.

Purpose of the Study:

• To assess the accuracy and comprehensiveness of LLM responses to diabetic retinopathy (DR) questions.
• To evaluate the readability and self-correction capabilities of LLMs in an ophthalmological context.

Main Methods:

• A cross-sectional study analyzed 252 responses from six LLMs to 42 DR-related questions.
• Consultant ophthalmologists independently graded responses for accuracy and comprehensiveness.
• Readability and self-correction capabilities were statistically analyzed.

Main Results:

• LLM responses varied in word and character counts, with significant differences in readability (ChatGPT-3.5 being least readable).
• Response accuracy was generally high, with ChatGPT-4.0 achieving 97.6% good ratings.
• Self-correction prompts significantly improved average accuracy scores from 6.4 to 7.5.

Conclusions:

• LLMs demonstrate potential for accurate and comprehensive answers regarding diabetic retinopathy (DR).
• Readability needs enhancement before clinical integration in ophthalmology.
• Continuous validation is crucial to ensure the reliability of LLM-generated information.