Comparing Performance of Large Language Model-Based Tools on Patient-Driven Glaucoma Inquiries

Area of Science:

• Artificial Intelligence in Medicine
• Ophthalmology
• Health Informatics

Background:

• Large language models (LLMs) are increasingly used by patients seeking online medical information for conditions like glaucoma.
• Understanding the performance variations among LLMs is crucial for guiding patients to reliable resources for glaucoma care.

Purpose of the Study:

• To evaluate and compare the performance of different LLMs in answering patient-related glaucoma questions.
• To assess the accuracy, comprehensiveness, quality, and readability of responses generated by AI chatbots.

Main Methods:

• A cross-sectional study involved prompting GPT-4o, GPT-4o Mini, Gemini Pro, and Gemini Flash with seven patient-generated glaucoma questions.
• Responses were assessed by ophthalmologists for accuracy, comprehensiveness, and quality using Likert scales.
• Readability was measured using Flesch-Kincaid Grade level, and semantic similarity was assessed using BERT Scores.

Main Results:

• GPT-4o demonstrated significantly higher accuracy, comprehensiveness, and quality than Gemini Pro.
• GPT-4o Mini also showed superior comprehensiveness and quality compared to Gemini Pro.
• No significant differences in readability were observed, and most LLM responses were semantically similar.

Conclusions:

• GPT models (GPT-4o and GPT-4o Mini) outperform Gemini Pro in providing high-quality, comprehensive information on glaucoma.
• AI chatbots show significant potential for delivering medically relevant knowledge to patients managing their eye health.
• These results can inform patients and healthcare providers about the most effective AI tools for glaucoma-related inquiries.