Large language model chatbots for patient-facing erectile dysfunction information: a cross-sectional evaluation of safety, accuracy, empathy, information quality and readability

Generative artificial intelligence chatbots are increasingly used for health information, including sensitive conditions, such as erectile dysfunction (ED), but their safety and quality remain uncertain. To compare six large language model chatbot interfaces for ED information across safety, accuracy, empathy, information quality, source-related transparency, readability and response length.

Thirty-eight standardized ED questions were submitted once to six chatbot interfaces, yielding 228 responses. Five independent raters assessed Safety, Accuracy, Empathy, DISCERN, Ensuring Quality Information for Patients, the Journal of the American Medical Association benchmark and Global Quality Score. Readability and response length were also evaluated. Between-interface differences were tested using Cochran's Q or Friedman tests.

Twenty-six responses (11.4%) were safety-flagged. Safety-flag frequency differed across interfaces (Q = 11.765, P = 0.038), but the effect size was small (0.062; 95% CI, 0.026-0.148), and no pairwise difference remained significant after multiplicity correction. Accuracy was generally high (median 4.00/5 for all interfaces), but differed slightly overall (P = 0.010, Kendall's W = 0.080). Empathy (P < 0.001, W = 0.160), DISCERN (P = 0.003, W = 0.508), Ensuring Quality Information for Patients (P < 0.001, W = 0.576), and Journal of the American Medical Association benchmark scores (P < 0.001, W = 0.888) differed significantly, whereas Global Quality Score did not (P = 0.146). Readability and response length also differed (all P < 0.001).

Large language model chatbots generally provided accurate ED-related information, but performance varied across clinically relevant dimensions. They may support information seeking, but should not replace individualized clinical assessment.