Performance of popular large language models in glaucoma patient education: A randomized controlled study

Area of Science:

• Artificial Intelligence in Medicine
• Ophthalmology
• Health Informatics

Background:

• Large language models (LLMs) are transforming knowledge acquisition but their use in patient education for conditions like glaucoma is underexplored.
• Effective patient education is crucial for managing chronic eye diseases such as glaucoma.

Purpose of the Study:

• To compare the performance of four common LLMs (Qwen, Baichuan 2, ChatGPT-4.0, PaLM 2) for glaucoma patient education.
• To evaluate the suitability of LLMs for providing glaucoma-related information to patients.

Main Methods:

• Initial evaluation by senior ophthalmologists scoring LLM responses to common glaucoma questions.
• Readability assessment using the Chinese Readability Platform.
• Real-world clinical testing with 29 glaucoma patients and attending ophthalmologists evaluating chatbot answers on correctness, completeness, readability, helpfulness, and safety.

Main Results:

• Baichuan 2 and ChatGPT-4.0 initially outperformed Qwen and PaLM 2.
• ChatGPT-4.0 exhibited higher recommended reading age and difficulty scores.
• In clinical testing, Baichuan 2 and ChatGPT-4.0 showed similar, high performance, with no significant differences between patient and ophthalmologist scores.

Conclusions:

• Baichuan 2 and ChatGPT-4.0 are identified as leading LLMs for glaucoma patient education.
• These models offer promising, viable tools for enhancing patient understanding and engagement in glaucoma care.