Leveraging artificial intelligence to streamline documentation and support patient-centered gynecologic oncology...

This study aims to evaluate provider-perceived accuracy, completeness, and clinical utility of ChatGPT-generated assessment and plans for outpatient gynecologic oncology encounters. A secondary aim is to identify prompt elements associated with accurate outputs to optimize the reliability of artificial intelligence (AI) integration.

Six clinical scenarios across a spectrum of outpatient gynecologic oncology visits were entered into ChatGPT (GPT-4) using a standardized prompt to generate electronic health record assessment and plan sections. Outputs were reviewed by gynecologic, radiation, and medical oncology faculty at a National Cancer Institute-designated tertiary referral center. Reviewers were blinded to other's evaluations. Accuracy was rated on a 5-point Likert scale and completeness on a 3-point Likert scale. Reviewers also indicated whether they would consider clinical use. Descriptive statistics summarized responses.

Fourteen faculty evaluators (response rate 77.8%) completed reviews, providing 84 accuracy ratings and 84 completeness ratings. Median accuracy was 4.0 (IQR 4.0-5.0), with 91.7% of ratings "mostly correct" or higher. Median completeness was 2.0 (IQR 2.0-3.0), with 91.7% of ratings "adequate" or "comprehensive." 80.6% of responses indicated willingness to use the AI-generated outputs in clinical practice. Analysis of the prompt-development process and reviewer feedback also identified key elements for generating reliable outputs such as role specification and guideline parameters.

Chat-GPT is capable of producing accurate, complete, and clinically useful outpatient gynecologic oncology plans as rated by oncology faculty. Specific prompt engineering to reflect subspecialty clinical context and established treatment standards may improve the consistency and reliability of AI-generated documentation.