Large Language Models Use in Dry Eye Disease: Perplexity AI versus ChatGPT4

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Informatics

Background:

• Dry eye disease (DED) is a prevalent condition affecting millions globally.
• The integration of artificial intelligence (AI) in healthcare, specifically large language models (LLMs), offers potential for clinical and research applications.
• Evaluating the efficacy and limitations of LLMs in specialized medical fields like DED is crucial.

Purpose of the Study:

• To compare the utility of two prominent LLMs, ChatGPT4 and Perplexity AI, in addressing queries relevant to dry eye disease (DED) within clinical and research contexts.
• To assess the performance of LLMs in generating patient-centered information and research ideas for DED.

Main Methods:

• Ocular surface experts developed 12 prompts, comprising 10 patient-focused questions and 2 research ideas for DED.
• Responses from ChatGPT4 and Perplexity AI were evaluated by experts using a standardized grading system (1-4) assessing accuracy, compassion, comprehensiveness, professionalism, humanness, and overall quality.
• Mean scores for each response characteristic were compared between the two LLMs.

Main Results:

• Both ChatGPT4 and Perplexity AI achieved comparable overall quality scores for patient-related DED queries (mean scores 2.6 and 2.7, respectively).
• ChatGPT4 demonstrated superior performance in generating DED-related research ideas compared to Perplexity AI (mean score 3.4 vs. 2.6).
• Perplexity AI's citations were web-based and not evidence-based, and inter-rater reliability for response characteristics was low for both models.

Conclusions:

• LLMs like ChatGPT and Perplexity AI show comparable utility for patient-related queries in DED, suggesting a potential role in patient education and counseling under supervision.
• These LLMs are currently not sufficiently advanced for generating novel research ideas or conducting comprehensive literature reviews in the field of DED.
• Further development and validation are needed to enhance the reliability and evidence-based nature of LLM responses in specialized medical domains.