Comparative quality, readability, and factual reliability of four publicly accessible chatbot systems for heart failure patient education: a cross-sectional evaluation

Large language models (LLMs) are increasingly used for health information delivery, but comparative evidence on their quality, readability, and factual reliability in heart failure (HF) education remains limited.

To compare four publicly accessible chatbot systems under prespecified access and retrieval configurations in answering common heart failure-related questions.

We evaluated 108 standardized English HF questions, yielding 432 responses. Content quality was assessed using modified DISCERN (mDISCERN) and Ensuring Quality Information for Patients (EQIP), readability was assessed using established English-language readability indices, including the Simple Measure of Gobbledygook (SMOG), Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Automated Readability Index (ARI), Gunning Fog Index (GFOG), and Coleman-Liau Index (CLI), and factual reliability was evaluated by auditing all responses against the 2022 AHA/ACC/HFSA guideline and relevant clinical evidence. Continuous outcomes were analyzed using linear mixed-effects models. Hallucination outcomes were defined as clinically meaningful statements that were inconsistent with guidelines, unsupported by evidence, overgeneralized beyond the relevant population, or potentially misleading for patient education, and were analyzed using Wilson 95% confidence intervals (CIs), Cochran's Q test, and generalized estimating equation (GEE) logistic regression.

DeepSeek generally showed the most favorable content-quality profile across the evaluated domains. In the primary readability analysis, Perplexity generated the most complex text, whereas sensitivity analysis using a second calculator showed partially different rankings. Hallucinations were identified in 6/108 (5.56%) DeepSeek responses, 8/108 (7.41%) ChatGPT responses, 10/108 (9.26%) Copilot responses, and 16/108 (14.81%) Perplexity responses. Paired binary analyses showed significant differences among chatbot systems (Cochran's Q p = 0.002; GEE p = 0.006).

Under the tested conditions and evaluated model versions, DeepSeek generally showed the most favorable content-quality profile and the lowest observed hallucination rate. However, hallucinations occurred across all systems, and pairwise factual-reliability differences were mainly driven by the higher hallucination rate of Perplexity. Guideline-aligned verification and clinician oversight remain necessary before patient-facing use.