Artificial Intelligence Performs Well for Patient-Level Education of Benign Anorectal Conditions

Area of Science:

• Colorectal Surgery
• Artificial Intelligence in Healthcare
• Medical Informatics

Background:

• Growing patient use of AI chatbots like ChatGPT for health information.
• Limited evaluation of AI chatbot performance for specific surgical topics.
• Need to assess AI accuracy for benign anorectal conditions.

Purpose of the Study:

• Evaluate ChatGPT's (GPT-4) accuracy for patient-level information on hemorrhoids, anal fissures, and pruritus ani.
• Compare ChatGPT's performance against established medical literature.
• Assess the suitability of AI chatbots for colorectal surgery patient education.

Main Methods:

• Prospective, single-institution study.
• Utilized the modified EQIP (mEQIP) tool for evaluation.
• Scored by three independent medical reviewers.

Main Results:

• ChatGPT achieved a median score of 61% (22/36) across all conditions.
• Performance was consistent for hemorrhoids (24/36), anal fissures (22/36), and pruritus ani (20/36).
• Highest scores in Content domain, weakest in Identification, citing non-evidence-based treatments and broken links.

Conclusions:

• ChatGPT offers accurate and reliable patient-level information for benign anorectal diseases.
• AI chatbot performance is comparable to other online health resources.
• Potential for AI chatbots in patient education, with recommendations for enhanced source documentation and visual aids.