Large Language Models vs. Professional Resources for Post-Treatment Quality-of-Life Questions in Head and Neck...

Area of Science:

• Health Informatics
• Artificial Intelligence in Healthcare
• Patient Education

Background:

• Patients increasingly use large language models (LLMs) like ChatGPT, Gemini, and Claude for health concerns.
• The quality, understandability, actionability, and empathy of LLM-generated health information are not well-established.
• This study compares LLM-generated patient information against professional resources.

Purpose of the Study:

• To evaluate the readability, understandability, actionability, and empathy of LLM responses.
• To compare LLM-generated content with information from professional healthcare resources.

Main Methods:

• A comparative cross-sectional study using 14 patient-style questions derived from validated quality-of-life instruments.
• Responses were generated by three LLMs (ChatGPT-4o, Gemini 2.5 Pro, Claude Sonnet 4) and two professional sources.
• Evaluation involved the Patient Education Materials Assessment Tool (PEMAT), DISCERN, Empathic Communication Coding System (ECCS), and readability metrics (Flesch Reading Ease, Flesch-Kincaid Grade Level).

Main Results:

• No significant differences were found in quality (DISCERN), understandability, actionability (PEMAT), or empathy (ECCS) between LLMs and professional resources.
• Professional resources demonstrated superior readability compared to LLM-generated responses.
• Statistical analysis included one-way ANOVA and Tukey's HSD test.

Conclusions:

• LLMs produce patient information comparable to professional resources in key quality aspects.
• Readability is a significant limitation of LLM-generated content, often necessitating simplification for health literacy.
• Further refinement of LLMs is needed to meet recommended health-literacy standards for patient education.