Artificial intelligence compared with human-derived patient educational materials on cirrhosis

Area of Science:

• Medical Education
• Health Informatics
• Artificial Intelligence in Healthcare

Background:

• Patient education materials (PEMs) are crucial for managing chronic conditions like cirrhosis.
• Evaluating the quality of AI-generated PEMs against traditional human-derived materials is essential.

Purpose of the Study:

• To compare the readability, understandability, actionability, and accuracy of AI-generated cirrhosis PEMs with human-derived materials.
• To assess patient and clinician perceptions of AI-generated PEMs.

Main Methods:

• Four large language model chatbots generated cirrhosis PEMs using a standardized prompt.
• PEMs were evaluated for readability (Flesch-Kincaid, SMOG) and scored by patients/caregivers and hepatologists on understandability and actionability.
• Medical accuracy was also assessed by hepatologists.

Main Results:

• Most PEMs, both AI and human-derived, exceeded the target sixth-grade reading level.
• All materials were understandable, but only human-derived material was consistently actionable.
• Neither patients nor clinicians could reliably distinguish between AI and human-generated content.

Conclusions:

• AI-generated PEMs demonstrate comparable readability, understandability, and accuracy to human-derived materials for cirrhosis.
• Readability, grade level, and actionability require improvement across all PEMs.
• AI-generated PEMs hold potential for clinical application, warranting further investigation.