Improving Pelvic Floor Disorder Education: A Second Pilot of a Retrieval-Augmented AI Chatbot Model

Artificial intelligence (AI) tools such as ChatGPT are increasingly used by patients to seek health information, yet concerns remain regarding the accuracy and reliability of AI-generated content. Integrating evidence-based resources into these models may improve their educational effectiveness. In a previous pilot study, a retrieval-augmented ChatGPT model was found to outperform a standard ChatGPT model under controlled conditions, highlighting the need for further testing in pragmatic clinical contexts. The objective of this second pilot was to evaluate whether a retrieval-augmented AI model improves response quality and usability for patient-representative urogynecology queries, as rated by experts using validated instruments.

We developed a retrieval-augmented ChatGPT model grounded in the American Urogynecologic Society's (AUGS) patient education materials. Urogynecology specialists were recruited through professional networks and submitted patient-representative questions. Responses were assessed across six domains using the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool, with usability measured by the System Usability Scale (SUS). Quantitative data were analyzed descriptively, and qualitative feedback was examined thematically.

Twenty-two questions were posed, with 11 out of 20 respondents providing complete responses. The median QAMAI score was 28 (IQR 24-29.8) and the mean SUS score was 81.8 ± 11.2. Respondents highlighted patient-friendly language, ease of use, and integration of AUGS links as strengths, while noting citation consistency, clinical depth, and comprehensiveness as areas for improvement.

A retrieval-augmented ChatGPT model trained on AUGS materials generated high-quality responses with excellent usability. Though not a substitute for clinician counseling, such models may supplement provider-patient communication.