Quality of patient information on interstitial cystitis from artificial intelligence chatbots

To evaluate the quality (DISCERN), understandability and actionability (Patient Education Materials Assessment Tool for Printable Materials [PEMAT-P]), readability (Flesch-Kincaid), and misinformation of patient-facing information on interstitial cystitis generated by four publicly available artificial intelligence (AI) chatbots: ChatGPT-4.0, Perplexity, ChatSonic, and Bing AI.

A total of 10 queries derived from Google Trends and Hopkins Medicine content were submitted to each chatbot. Responses were evaluated by two blinded reviewers using validated tools: the DISCERN instrument (reliability/quality), PEMAT-P (understandability/actionability), and Flesch-Kincaid Grade Level (readability). Word count and citation inclusion were also recorded.

Across chatbots, information quality was moderate with a median (interquartile range [IQR]) DISCERN score of 3/5 (2-3), with Perplexity performing best and Bing AI worst. Understandability was moderate (median [IQR] PEMAT-P score 75% [66.7-83.3%]), highest for ChatSonic with Hopkins Medicine-derived prompts and lowest for ChatGPT with Google Trends inputs. Actionability was consistently poor (median [IQR] score 40% [20-60%]), with ChatSonic performing best and Bing AI lowest. Responses averaged 256 words and college-level readability (median [IQR] Flesch-Kincaid score 25.4 [20.89-28.50]) across all platforms, limiting accessibility. Misinformation was minimal across all platforms. Chatbots referencing clinically curated prompts (Hopkins Medicine) scored higher in understandability and completeness than those responding to public search trends.

Artificial intelligence chatbots offer generally accurate and understandable information about interstitial cystitis but lack actionable guidance and generate content at reading levels above typical patient comprehension. Enhancing readability, actionability, and personalisation may increase their utility as adjunct tools for patient education in functional urology.