Evaluation of Glaucoma Treatment Information on Social Media Using Large Language Models

Area of Science:

• Ophthalmology
• Medical Informatics
• Artificial Intelligence in Healthcare

Background:

• Social media platforms are increasingly used for health information dissemination.
• The quality of health information, particularly for complex conditions like glaucoma, varies significantly on these platforms.
• Evaluating the accuracy, readability, utility, and educational value of such content is crucial.

Purpose of the Study:

• To assess the quality of glaucoma treatment information shared on social media.
• To compare the evaluation of social media content quality by glaucoma experts and a large language model (LLM).

Main Methods:

• Five glaucoma experts rated social media posts on glaucoma treatment using a 5-point Likert scale across four domains.
• A reference standard for content quality was established based on expert consensus.
• A large language model (GPT-4) was prompted to evaluate the same posts using identical instructions.
• Agreement between experts and the LLM was analyzed using statistical measures (Cohen's kappa).

Main Results:

• Glaucoma experts judged only 40% of social media posts on glaucoma treatment favorably.
• The LLM (GPT-4) rated content favorably in 77% of cases, significantly more often than experts (P=0.017).
• Moderate agreement (κ=0.421) was observed between the LLM and experts, primarily when content was rated favorably.

Conclusions:

• While LLMs show moderate agreement with expert opinion on social media content quality, they are less discerning of low-quality information.
• AI-based systems may overestimate the quality of health information found on social media compared to human experts.
• Further research is needed to refine AI tools for accurately assessing medical information online.