A Comparative Study of Responses to Retina Questions from Either Experts, Expert-Edited Large Language Models, or...

Area of Science:

• Ophthalmology and Artificial Intelligence
• Medical Informatics
• Natural Language Processing in Healthcare

Background:

• Patient education is crucial in ophthalmology.
• Large Language Models (LLMs) offer potential for generating patient information.
• Assessing the quality, empathy, and safety of LLM-generated content is essential.

Purpose of the Study:

• To compare the quality, empathy, and safety of responses to common retina patient questions generated by LLMs, human experts, and expert-edited LLMs.
• To evaluate the efficiency of expert editing of LLM responses.

Main Methods:

• A randomized, masked, multicenter study involving 13 retina specialists.
• Experts created original responses and edited LLM-generated responses (Expert + AI).
• Multiple LLMs (ChatGPT-3.5, ChatGPT-4, Claude 2, Bing, Bard) generated responses.
• Responses were evaluated by other experts for quality, empathy, and safety metrics.

Main Results:

• Expert + AI responses achieved the highest mean quality score.
• GPT-3.5 and Expert + AI led in empathy scores.
• Expert-edited LLM responses significantly outperformed expert-created responses in quality and empathy.
• Significant time savings were observed with the Expert + AI approach.

Conclusions:

• LLM responses, particularly when expert-edited, are comparable to human experts in quality, empathy, and safety for retina patient questions.
• The findings support further investigation into using LLMs in clinical ophthalmology settings.
• Expert-refined LLMs present a viable tool for enhancing patient education and information dissemination.