Reproductive health and ChatGPT: an evaluation of AI-Generated responses to commonly asked abortion questions

Recent assessments of ChatGPT in relation to a variety of pregnancy-related questions have shown mixed results. Rapidly evolving rules and regulations in the USA have led to a confusing abortion landscape, making up-to-date and evidence-based abortion information essential to those considering an abortion. The purpose of this study was to evaluate ChatGPT as a source of information for commonly asked medication and procedural abortion questions by performing a qualitative analysis. We queried ChatGPT-3.5 on ten fact-based abortion questions and ten clinical scenario abortion questions. Query responses were graded by three complex family planning physicians to be 'acceptable' or 'unacceptable' and 'complete' or 'incomplete'. The responses were then compared to evidence-based research published by the American College of Obstetricians and Gynaecologists (ACOG), the Society of Family Planning (SFP), PubMed-indexed evidence, as well as physician clinical experience. In our assessment, a grade of acceptable was given to 65% of responses, however a grade of complete was only given to 8% of responses. In general, fact-based questions were more accurate than clinical questions. Our analysis of ChatGPT suggested it can regurgitate facts found online, but it still lacks the ability to provide understanding and context to clinical scenarios that clinicians are better equipped to navigate.