Reliability, Readability, and Safety of Generative AI Chatbots for Osteoporosis Health Counseling: A Multidimensional...

Generative artificial intelligence chatbots are increasingly used for health information, but their reliability, readability, and safety in osteoporosis counseling remain uncertain. This study compared four widely available chatbots under standardized conditions.

Twenty-five standardized osteoporosis questions were submitted once to ChatGPT-5, Gemini 2.5 Pro, Copilot, and Perplexity Pro, generating 100 responses. Two model-blinded specialists evaluated the final responses using DISCERN, EQIP, the Global Quality Scale, JAMA benchmarks, and the NOHARM framework. Six automated indices assessed readability. Because each question was answered by all four models, outcomes were compared using Friedman tests, followed by paired Wilcoxon signed-rank tests with Holm adjustment.

DISCERN scores differed across models (p < 0.001; Kendall's W = 0.82, 95% CI 0.76-0.89). Median scores were 55.0 for Perplexity Pro, 54.0 for Copilot, 44.0 for Gemini 2.5 Pro, and 42.0 for ChatGPT-5. JAMA benchmark scores did not differ significantly (p = 0.63). All six readability indices differed across models (all p < 0.001). Perplexity Pro had the highest median Flesch-Kincaid Grade Level score at 16.18, whereas Copilot had the lowest at 9.49; all models exceeded the sixth-grade benchmark. The qualitative safety assessment also identified clinically relevant omissions and an unsupported claim.

In this set of osteoporosis questions, retrieval-enabled systems achieved higher scores on selected information-quality measures but produced more difficult text. These findings are limited to the tested models, prompts, and evaluation period and do not establish a universal effect of retrieval augmentation.