Evaluation and Comparison of Large Language Models in Answering Frequently Asked Patient Questions on Glaucoma

ChatGPT-o1 and DeepSeek-R1 demonstrated similar performance in answering frequently asked patient questions. However, limitations in reliability and high reading levels underscore the need for greater oversight and refinement to improve their role in patient education.

This study evaluated and compared the reliability and readability of responses generated by ChatGPT-o1 and DeepSeek-R1 to frequently asked patient questions (FAQs) on glaucoma. The findings aim to elucidate the utility of each model in supporting effective patient education.

Glaucoma FAQs were obtained from the first 10 questions on Google's "People Also Ask" section. Topics included foundational knowledge, treatment description, lifestyle, and prognosis. Each question was input into ChatGPT-o1 and DeepSeek-R1, and responses were independently graded for reliability by two glaucoma specialists using a 4-point scale (1=excellent, no clarification needed; 4=unsatisfactory, substantial clarification needed). For further comparison, the top two Google search result webpages for the same queries were similarly graded. Readability grade levels were assessed using validated metrics.

ChatGPT-o1 and DeepSeek-R1 received comparable median reliability scores (2.3 [1.6-3] vs. 2.8 [2.1-3], P=0.08), which were similar to scores of top Google webpages (2.3 [1.8-2.7], P=0.18). Both chatbots struggled with questions on foundational knowledge (i.e. falsely suggesting that glaucoma may cause tunnel vision) and lifestyle (i.e. offering unsubstantiated advice on chugging water, dietary recommendations, and screen time). Responses generated by both models were presented at the 8th-12th grade reading level.

ChatGPT-o1 and DeepSeek-R1 responses were comparable, demonstrating notable limitations in accuracy. Reading levels similarly exceeded average U.S. adult literacy, posing challenges for patient comprehension. Rigorous oversight and targeted refinement of LLMs is necessary to enhance the reliability and accessibility of content for patient education in glaucoma.