Artificial Intelligence Performs Well for Patient-Level Education of Benign Anorectal Conditions

BackgroundChatGPT, the most widely recognized AI chatbot, has been increasingly utilized by patients for health care education, but its performance and knowledge base have not been evaluated for colorectal surgery topics. We hypothesize ChatGPT can provide accurate patient-level information for benign anorectal diseases.MethodsWe performed a single-institution prospective study evaluating OpenAI's GPT-4 chatbot against verified online medical literature using the modified EQIP (mEQIP) for hemorrhoids, anal fissures, and pruritus ani. Scoring was performed by three independent educated reviewers.ResultsChatGPT had a median overall score of 22/36 (61%) across all topics. It performed similarly for all topics: 24 for hemorrhoids, 22 for fissures, and 20 for pruritus (P = 0.15). ChatGPT was strongest within the mEQIP Content domain for all topics (P = 0.001). It performed weakest within the mEQIP Identification domain. It primarily lost points for recommending non-evidence-based treatments, lack of citations and visual aids, and generating broken source links.DiscussionChatGPT can provide accurate, reliable information at patient understanding level and is comparable to other validated online information for benign anorectal pathologies. It could improve on mEQIP performance by improving source documentation and visual aid capability, but it remains a promising patient resource with an intuitive user interface.