Evaluating the Quality of AI-Written Scenarios for Virtual Oral Surgical Board Preparatory Examination

The objective of this study was to evaluate whether ChatGPT can generate level-appropriate clinical scenarios that are suitable for use in oral board preparatory examination (mock oral exam) for senior surgical residents.

This was a prospective, blinded study in which AI-written and faculty-written scenarios were reviewed, randomized and used for testing in virtual mock oral exam. Both faculty examiners and test-taking residents were blinded to the true authorship of the scenarios. After the examination, participants completed a survey evaluating the complexity of each scenario and their perceptions of its authorship.

The study was conducted at Washington University in St. Louis (WashU), an academic medical center located in St. Louis, Missouri. The participating institutions also included Saint Louis University (SLU).

Study participants included twenty-five senior general surgery residents (PGY4 and PGY5) and twenty faculty examiners from WashU and SLU, who took part in virtual mock oral examination. Post-exam surveys were completed both residents and faculty.

Faculty rated most AI-written and faculty written scenarios as "level-appropriate" in terms of both the quality of the text and the degree of complexity. Similarly, when residents were asked to identify the most difficult scenarios, they selected both AI- and faculty-written scenarios at comparable rates. Notably, both faculty and residents struggled to correctly distinguish the origin of the scenarios, with frequent misidentification across both groups.

AI-written clinical scenarios were comparable to faculty-written scenarios in terms of complexity and appropriateness for senior surgical residents when used in a virtual mock oral board examination, highlighting the potential utility of AI-based tools in oral board preparation and surgical education.