Quality of Information About Kidney Stones from Artificial Intelligence Chatbots

Introduction: Kidney stones are common and morbid conditions in the general population with a rising incidence globally. Previous studies show substantial limitations of online sources of information regarding prevention and treatment. The objective of this study was to examine the quality of information on kidney stones from artificial intelligence (AI) chatbots. Methods: The most common online searches about kidney stones from Google Trends and headers from the National Institute of Diabetes and Digestive and Kidney Diseases website were used as inputs to four AI chatbots (ChatGPT version 3.5, Perplexity, Chat Sonic, and Bing AI). Validated instruments were used to assess the quality (DISCERN instrument from 1 low to 5 high), understandability, and actionability (PEMAT, from 0% to 100%) of the chatbot outputs. In addition, we examined the reading level of the information and whether there was misinformation compared with guidelines (5 point Likert scale). Results: AI chatbots generally provided high-quality consumer health information (median DISCERN 4 out of 5) and did not include misinformation (median 1 out of 5). The median understandability was moderate (median 69.6%), and actionability was moderate to poor (median 40%). Responses were presented at an advanced reading level (11th grade; median Flesch-Kincaid score 11.3). Conclusions: AI chatbots provide generally accurate information on kidney stones and lack misinformation; however, it is not easily actionable and is presented above the recommended reading level for consumer health information.