Performance evaluation of mainstream large language models in cataract science popularization Q&A: a comparative...

Cataract is one of the main causes of visual impairment and reversible blindness worldwide, and it mainly affects the elderly population. Clinically accurate and sufficiently readable patient education materials play a crucial role in this regard. With the rapid development of large language models (LLMs), patients are increasingly obtaining health information generated by artificial intelligence (AI); however, the reliability of online medical information is often questionable. This study systematically evaluated the readability, quality, and educational suitability of mainstream LLMs when answering questions related to cataract.

Five mainstream LLMs - Doubao, DeepSeek, Wenxin Yiyan, Tongyi Qianwen, and GPT-5 - were evaluated based on their responses to 20 frequently asked questions (FAQs) for cataract patient, which covered five subject categories. Text readability was assessed through multiple indicators, including the Coleman-Liau Index (CL), Linsear Write (LW), Automated Readability Index (ARI), Simple Measure of Gobbledygook (SMOG), Gunning Fog Index (GFOG), Flesch Reading Ease Score (FRES), and Flesch-Kincaid Grade Level (FKGL). Information quality and educational suitability were evaluated using the Global Quality Score (GQS) and the Chinese version of the Patient Education Material Readability Assessment Tool (c-PEMAT-P). Differences between groups were compared using one-way ANOVA and Kruskal-Wallis tests, with correlation analyses exploring relationships among indicators.

There were significant differences among LLMs in terms of readability, information quality, and educational suitability (all p < 0.05). GPT-5 had the highest c-PEMAT-P and GQS scores, however, several readability difficulty indices of GPT-5 and Tongyi Qianwen were also higher. There were significant differences in readability among different content categories. The postoperative management and risk/prevention topics tended to have better educational suitability, whereas surgical diagnosis, treatment, and preoperative management were more difficult to read. Correlation analysis demonstrated that the correlation between the quality indicators and the readability-related indicators is generally weak.

In terms of generating educational materials for cataract patient, LLMs have potential, but the output quality varies depending on the model and the topic. GPT-5 performs best in terms of overall quality and educational suitability, but its readability is not always at an ideal level. Model selection has a crucial impact on the quality and educational suitability of the information, while the content topic mainly affects the language complexity. Therefore, for the responsible use of LLMs in cataract patient education, the cataract education materials generated by LLMs need to undergo expert review, readability optimization, and patient-centered validation before clinical application.