Artificial Intelligence Chatbots for Dysphagia Patient Education: A Multi-Center International Expert Evaluation

Area of Science:

• Artificial Intelligence in Healthcare
• Medical Informatics
• Patient Education

Background:

• Growing patient reliance on AI chatbots for health information.
• Lack of validation for AI-generated content on complex conditions like dysphagia.
• Need for comparative analysis of leading AI models in dysphagia patient education.

Purpose of the Study:

• To compare the accuracy, clarity, and readability of AI-generated dysphagia patient education content.
• To evaluate leading large language models (LLMs) head-to-head for dysphagia information.
• To assess the accessibility of AI-generated health information for patients.

Main Methods:

• Submitted 46 validated dysphagia questions to ChatGPT-4.0 and Claude 3.7.
• Evaluated responses by 10 international experts for accuracy, clarity, and misinformation.
• Assessed content readability using Flesch Reading Ease, Flesch-Kincaid Grade Level, and SMOG Index.
• Utilized Wilcoxon signed-rank tests for statistical comparisons.

Main Results:

• No significant differences in scientific accuracy or clarity between ChatGPT and Claude.
• Both AI models generated content with readability levels significantly exceeding recommendations (SMOG indices ~15-17 years).
• Claude produced a higher proportion of misinformation-free responses (19.6% vs. 6.5%).

Conclusions:

• Leading AI chatbots provide accurate but overly complex dysphagia information.
• Shared limitations in AI training data contribute to readability challenges.
• Mandatory readability optimization of AI-generated patient education is essential for clinical implementation.