Large Language Models Use in Dry Eye Disease: Perplexity AI versus ChatGPT4

To compare the utility of two large language models (LLM) in dry eye disease (DED) clinics and research.

Trained ocular surface experts generated 12 prompts for queries commonly asked by DED patients (n = 10 questions) and research ideas for DED (n = 2). Responses of two LLM models, ChatGPT4 and Perplexity AI, were graded by them using a standardized grading system (1 = needs improvement, 2 = fair, 3 = good, and 4 = excellent) evaluating the response accuracy, compassion, comprehensiveness, professionalism, humanness, and overall quality of each response. The mean scores of the grades from each expert for each response were compared.

The 10 clinical DED prompts received similar overall mean quality grades for the responses with ChatGPT (mean grade score = 2.6) and Perplexity AI (2.7). The mean grade scores for the response characteristics (accuracy, compassion, professionalism, humanness, and succinctness) varied between the experts for each question (range 2.2 to 3.1 for ChatGPT and 2.3 to 3.0 for Perplexity AI). ChatGPT4 generated DED-related research ideas better than Perplexity AI (mean 3.4 vs. 2.6). The source citations for responses by Perplexity AI were from web pages and were not evidence-based. There was slight or poor agreement between the reviewers' ratings for response characteristics generated by both LLMs.

Perplexity AI and ChatGPT performed similarly for patient-related queries on DED and could have a role in patient education. These LLMs could have a role in DED clinics for patient counseling but require supervision. The LLMs are not ready to generate dry-eye research ideas or perform literature searches for DED.