Evaluating Use of Chat-Generative Pretrained Transformer for Patient Education of Pelvic Floor Disorders

Pelvic floor disorders (PFDs) affect a growing number of women, who often avoid seeking medical attention. The emergence of digital technologies, including artificial intelligence and natural language processing, such as Chat-Generative Pretrained Transformer (ChatGPT) presents an opportunity for patients to access medical information online. This study evaluated ChatGPT's potential as a tool for patient education regarding pelvic floor disorders.

The primary outcome was to assess readability of ChatGPT responses to common patient questions about PFDs. Secondary outcomes included assessment of quality, accuracy, and reproducibility of ChatGPT responses.

ChatGPT 3.5 was queried for responses to common patient questions regarding PFDs. Queries were made using 2 separate accounts and on 2 separate dates. Each response was scored for readability by the Flesch Reading Ease score. Three independent reviewers evaluated ChatGPT outputs for quality and accuracy using a 5-point Likert Scale. Responses from separate accounts and dates were reviewed for reproducibility. Analysis was primarily descriptive.

The average readability for all PFDs was found to be 33.57/100, corresponding to a college reading level. The median quality ranged from 3 to 4, indicating responses were of "fair" to "good" quality. Expert reviewers found that content was on average 86.71% accurate with 25% variation across responses.

ChatGPT produces content at a college reading level, which is relatively accurate, of fair to good quality, and reproducible as judged by urogynecologists. Large language models may be a helpful patient education tool; however, further work remains to make ChatGPT accessible to individuals with low health literacy.