Benchmarking the readability, quality, and educational suitability of large language models in communicating...

Pertussis (whooping cough) is a highly contagious respiratory infection that continues to cause substantial morbidity and mortality, particularly among infants, and has re-emerged globally among adolescents and adults. Large language models (LLMs) are increasingly used for health communication and science popularization; however, evidence regarding their readability, quality, and educational suitability for disease-specific patient education remains limited. To date, no systematic evaluation has focused on LLM-generated pertussis health education materials.

This study aimed to systematically evaluate and compare the performance of five mainstream LLMs in generating pertussis-related science popularization content, with particular attention to readability, informational quality, and educational suitability.

A cross-sectional simulation study was conducted using 20 frequently asked pertussis-related questions covering five domains: basic knowledge, symptom presentation, diagnostic methods, treatment and management, and prevention and prognosis. On October 28, 2025, all questions were identically input into five publicly accessible LLMs. Text readability was assessed using seven classical indices. Two independent pharmacists performed blinded evaluations using the Chinese version of the Patient Education Materials Assessment Tool for print materials (C-PEMAT-P) and the Global Quality Score (GQS). Additionally, two independent clinical experts assessed the factual accuracy and guideline concordance of each LLM-generated response against the Chinese Pertussis Diagnosis and Treatment Guidelines (2024) using a 4-point scale. Inter-rater agreement was evaluated using Cohen's kappa coefficient.

ChatGPT, DeepSeek, and Doubao achieved significantly higher C-PEMAT and GQS scores than Wenxin Yiyan and Gemini (p < 0.001), indicating superior understandability, actionability, and overall quality. Median C-PEMAT scores across all models suggested generally acceptable accessibility for patient education. Regarding factual accuracy and guideline concordance, ChatGPT achieved the highest mean score. No harmful advice or direct guideline contradictions were identified in any model output. Correlation analyses showed weak associations between traditional readability metrics and GQS, whereas C-PEMAT demonstrated a moderate positive correlation with GQS (r = 0.34).

Mainstream LLMs show preliminary capability in generating pertussis-related health education materials, although substantial inter-model variability persists. Domain-specific patient education assessment tools better capture perceived informational quality than generic readability metrics. These findings support the cautious, assistive use of LLMs in health communication within a human-AI collaborative framework.