Evaluation of AI-Generated Surgical Decision Aids for Pelvic Floor Disorders

Decision aid tools are well-utilized resources in shared decision making for the treatment of pelvic floor disorders. With the improvement of large language models like ChatGPT, prior studies have started evaluating whether artificial intelligence can be used to create patient education materials. As treatment of pelvic floor disorders can be complicated and specific to individuals, we assessed the use of ChatGPT for patient decision making.

Our objective was to evaluate the understandability, reliability, accuracy, and readability of ChatGPT-generated decision aid tools for urogynecology surgical procedures.

In this cross-sectional study, 6 questions comparing 2 treatment options for common urogynecological conditions were developed by 6 urogynecologists and entered into ChatGPT to create decision aid tools. Patients and physicians evaluated understandability using a Patient Education Materials Assessment Tool. Physicians also evaluated reliability using a modified DISCERN (mDISCERN) instrument and accuracy using a 5-point Likert scale. Readability was evaluated using the Flesch-Kincaid Reading Ease score. Analysis of all scores was descriptive.

All decision aid tools received high patient and physician understandability. The tools had fair reliability with an average mDISCERN score of 26. Accuracy was <4 (unfavorable) on 2 of the decision aid tools, and the reading level required to read them was high overall.

ChatGPT-generated decision aid tools for urogynecology surgical counseling have high patient understandability, emphasizing their potential for being well-received by patients deciding on treatment. However, approximately 30% of the topics require improvements to accuracy, while all tools could benefit from improved reliability and readability.