Comparative Outcomes of ChatGPT Versus Faculty-Facilitated Simulation Design by Medical Students

Limited research describes generative artificial intelligence (AI) and large language models, such as ChatGPT, to support the design of health care simulation-based manikin scenarios (SBMS). This pilot study describes the design process and quality assessment of SBMS designed by medical students with ChatGPT generative AI support (AIG scenarios) compared to those designed by medical student interest groups (MSIG) with faculty guidance (MSIG scenarios).

Based on this medical school simulation center's scenario design template, we compared the time to completion of the scenario between five MSIG SBMS and five AIG SBMS on the same clinical topics. Three blinded simulation experts rated all randomized scenarios using the Simulation Scenario Evaluation Tool (SSET). We performed paired t tests to evaluate differences between the MSIG and AIG scenarios.

MSIG SBMS design required months of faculty-student collaboration, whereas AIG SBMS were completed in a mean of 37 minutes. AIG scenarios had significantly higher Scenario Materials and Resources SSET scores. We found no differences in other SSET elements or overall scores. Rater agreement across most categories was low. The Debriefing Plan element received the lowest scores in both groups.

This study supports the potential of simulation novices using ChatGPT to streamline health care SBMS design. AIG scenarios required less time to create and were comparable in quality to MSIG scenarios. By accelerating the initial scenario design process, AI offers an efficient tool for developing educational simulations, expanding opportunities for students to engage in scenario design as an educational methodology.