Accuracy and completeness of large language models in Epidemic keratoconjunctivitis Queries: A Comparative study

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Informatics

Background:

• Reliability of Large Language Models (LLMs) in specialized clinical domains like ophthalmology is not well-established.
• Epidemic keratoconjunctivitis (EKC), a contagious adenoviral eye infection, requires accurate diagnostic and treatment information.
• This study evaluates LLM performance in the context of EKC.

Purpose of the Study:

• To assess and compare the accuracy and completeness of ChatGPT and Meta AI responses to ophthalmology questions about EKC.
• To characterize LLM performance in a specific, high-contagion ophthalmic disease domain.
• To inform the potential use of LLMs in ophthalmology.

Main Methods:

• A cross-sectional study using 34 structured questions on EKC (etiology, clinical features, diagnosis, treatment, prognosis).
• Responses from ChatGPT and Meta AI were independently evaluated by two cornea specialists.
• Scoring used a five-point Likert scale for accuracy and completeness; inter-rater agreement was measured.

Main Results:

• Both LLMs demonstrated excellent performance, with high mean accuracy (4.87 for both) and completeness scores (4.90 for ChatGPT, 4.93 for Meta AI).
• Substantial agreement was found between expert raters (κ range 0.78-0.88).
• Meta AI showed slightly superior completeness, especially for diagnostic and symptom-related queries.

Conclusions:

• LLMs demonstrate robust performance and consistency in addressing EKC-related ophthalmic queries.
• Their capabilities support use as supplementary tools for knowledge retrieval, treatment explanation, and patient counseling in ophthalmology.
• Professional oversight is crucial when utilizing LLMs for clinical information on conditions like EKC.