Artificial Intelligence Chatbots and Narcolepsy: Friend or Foe for Patient Information?

Narcolepsy is a rare sleep disorder with a complex clinical picture, which may affect the daily functioning of patients. Artificial intelligence (AI) has emerged as a promising tool in healthcare, potentially offering valuable support to patients. However, its accuracy in specific medical domains remains inadequately assessed. This study aimed to evaluate and compare the accuracy, completeness, and readability of responses from ChatGPT, Gemini, and Perplexity to queries about narcolepsy.

This study was a cross-sectional study. A set of 28 common patient questions was selected and entered into the three chatbots. Responses were independently reviewed by three sleep physicians. Accuracy and completeness were rated on predefined 5-point and 3-point scales, respectively. Readability was evaluated using six validated formulas.

All chatbots showed median accuracy ranging from "more correct than incorrect" to "completely correct," with no significant performance differences. The topics with the lowest scores were "treatment and prognosis" for ChatGPT and Perplexity and "diagnosis" for Gemini. Gemini responses were significantly less complete compared to ChatGPT and Perplexity, with median completeness scores for ChatGPT and Perplexity ranging from "nearly complete" to "complete" and for Gemini ranging from "incomplete" to "nearly complete." All chatbots' responses required an advanced reading level, with Perplexity showing lower readability in five metrics.

Our findings highlight the potential of AI chatbots to deliver mostly accurate responses to narcolepsy-related queries. However, these tools have limitations, including text accessibility, as the readability of the responses did not align with the recommended standards for health information. Therefore, their use should be integrated with appropriate guidance from healthcare professionals to avoid potential misunderstandings.