Safety, accuracy, information quality, readability, and supportive communication of LLM-based chatbots in responses...

Recurrent aphthous stomatitis (RAS), commonly known as oral ulcers, is a frequent and painful oral mucosal condition that often prompts people to seek consumer health information before professional consultation. Although LLM-generated health advice has been studied in several clinical fields, multidimensional evidence focused specifically on RAS remains limited.

To compare the safety, accuracy, supportive communication/empathy, information quality and transparency, and readability of five publicly accessible LLM-based chatbots when responding to researcher-developed consumer-oriented prompts about RAS.

Thirty researcher-developed, consumer-oriented English prompts covering seven RAS domains were submitted once, in separate sessions, to ChatGPT, Copilot, Gemini, Doubao, and DeepSeek, yielding 150 single-turn responses. Safety was the primary clinical outcome; accuracy, supportive communication/empathy, and information-quality measures were secondary, and six formula-based readability indices were exploratory. Five reviewers scored each response independently before consensus adjudication. User-facing product labels, access conditions, prompts, reference checklists, complete responses, and statistical outputs are documented in the manuscript and supplementary materials.

Sixteen of 150 responses (10.7%) were consensus-coded as potentially unsafe or misleading. Safe-response rates ranged from 83.3% to 93.3%, with no detected between-chatbot difference (Cochran's Q = 2.957, P = 0.565). Potential safety concerns were observed in persistent or non-healing ulcers, cancer-related reassurance, home or traditional remedies, supplementation, pregnancy, and pediatric scenarios. No between-chatbot differences were detected for accuracy, supportive communication/empathy, DISCERN, EQIP, JAMA, or GQS. Five of six readability formulas detected differences, but only 31 of 150 responses (20.7%) met the prespecified FKGL target of grade 6 or below.

In this exploratory dataset, no between-chatbot differences were detected for safety, accuracy, supportive communication/empathy, or information-quality scores, whereas formula-dependent readability differences were observed. Non-significant findings do not establish equivalence, and readability formulas do not establish comprehension. The results apply only to the recorded user-facing configurations and single responses to 30 researcher-developed English prompts during the defined access period; they do not demonstrate readiness for autonomous clinical or patient-facing deployment.