Evaluating large language models for glaucoma counseling: A pilot study of ChatGPT and Google Gemini responses in...

Large language models (LLMs) such as ChatGPT and Google Gemini are increasingly used for patient education in various medical fields. However, their effectiveness in providing ophthalmology-related information in non-English languages remains underexplored. This study aimed to evaluate and compare the appropriateness of ChatGPT and Google Gemini responses to glaucoma-related questions presented in Traditional Chinese.

Twenty frequently asked glaucoma-related questions were translated into Traditional Chinese and categorized into four domains: disease understanding, diagnosis, treatment, and lifestyle. Responses were generated using ChatGPT (GPT-4) and Google Gemini. Three qualified ophthalmologists independently rated the appropriateness of each response using a 5-point Likert scale. Mean scores were calculated, and statistical comparisons were performed for overall and category-specific performance.

ChatGPT achieved a higher overall appropriateness score (4.32 ± 0.65) than Google Gemini (4.03 ± 0.66), with a statistically significant difference (P = 0.004). Across all four content domains, ChatGPT consistently outperformed Gemini, although the differences did not reach statistical significance. Both models demonstrated relatively lower performance in lifestyle-related questions.

Both ChatGPT and Google Gemini are capable of providing reasonably appropriate glaucoma-related information in Traditional Chinese. ChatGPT demonstrated superior overall performance. These findings support the potential role of LLMs in enhancing patient education in non-English settings, although further improvements in language localization and domain-specific accuracy are warranted.