Artificial Intelligence-Generated Patient Education Materials for Helicobacter pylori Infection: A Comparative...

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Health Communication

Background:

• Patient education is crucial for public awareness of Helicobacter pylori (H. pylori).
• Large language models (LLMs) present a transformative opportunity to enhance patient education.
• This study evaluates the quality of H. pylori patient educational materials (PEMs) generated by LLMs against physician-created content.

Purpose of the Study:

• To assess the quality of LLM-generated patient educational materials (PEMs) for H. pylori.
• To compare LLM-generated PEMs with physician-sourced PEMs in terms of accuracy, completeness, and comprehensibility.
• To evaluate PEMs in both English and Chinese at a sixth-grade reading level.

Main Methods:

• Physician and five LLMs (Bing Copilot, Claude 3 Opus, Gemini Pro, ChatGPT-4, ERNIE Bot 4.0) generated H. pylori PEMs.
• PEMs were assessed for accuracy, completeness, and comprehensibility by gastroenterologists and patients.
• Readability was evaluated using Flesch-Kincaid and Simple Measure of Gobbledygook scores.

Main Results:

• English PEMs from all sources showed acceptable accuracy and comprehensibility, but lacked completeness.
• Physician-sourced PEMs achieved the highest accuracy (5.60), while LLM English PEMs ranged from 4.00 to 5.40.
• Chinese LLM-generated PEMs generally scored lower in accuracy and completeness compared to English versions; no PEM met the target reading level.

Conclusions:

• LLMs demonstrate potential as tools for patient education.
• The accuracy and comprehensibility of LLM-generated PEMs are adequate, but completeness requires enhancement.
• Further optimization is necessary to address linguistic diversity and improve the overall feasibility of LLM-generated educational materials.