Comparative evaluation of large language models for esophageal cancer patient education: quality, reliability, readability, understandability, and actionability

Large language models (LLMs) are increasingly used as sources of health information. This study evaluated the quality, reliability, readability, understandability, and actionability of responses generated by three mainstream LLMs to esophageal cancer-related questions.

Thirty esophageal cancer-related questions covering six domains were identified from Google Trends, clinical practice guidelines, and social media and submitted to three LLMs, yielding 90 responses. Responses were evaluated using the Global Quality Scale (GQS), modified DISCERN (mDISCERN), Ensuring Quality Information for Patients (EQIP), Quality Assessment of Medical Artificial Intelligence (QAMAI), Patient Education Materials Assessment Tool (PEMAT), and six readability indices. Differences among the models were assessed using repeated-measures ANOVA or the Friedman test according to data distribution.

Crucial information missing was the most common content deficiency (8.9%). Overall GQS, mDISCERN, QAMAI, and EQIP scores were 4.00 (IQR: 4.00-4.00), 4.00 (IQR: 3.50-4.00), 22.00 (IQR: 21.50-25.00), and 71.88 (IQR: 62.50-75.00), respectively. DeepSeek performed best overall, followed by ChatGPT and Doubao. None of the three models met the recommended sixth-grade reading level, and all had PEMAT-A scores below the recommended 70% threshold. Readability was generally not significantly associated with quality or reliability, except for a few weak negative correlations.

LLMs demonstrated generally favorable quality and reliability when answering esophageal cancer-related questions, but readability and actionability remained limited. They may serve as supplementary tools for patients seeking esophageal cancer-related health information but should not replace professional medical consultation.