Artificial Intelligence in Simulation Design: The Effectiveness of a Customized Artificial Intelligence-based...

The creation of simulation scenarios is typically time-consuming and requires a high cognitive load. Artificial intelligence (AI) and large language models (LLMs) can generate text, including simulation scenarios, but the output quality can be very user-dependent. The goal of this study is to explore if a customized version of ChatGPT can create simulation scenarios that are comparable to those written by humans.

In this observational cross-sectional study, simulation content experts scored 8 scenarios covering 4 pediatric emergency medicine (PEM) topics with one version written by health care educators and published on MedEdPORTAL, and another by a custom ChatGPT. Experts were blinded to the source of each case. The scenarios were assessed in 3 ways: an objective score from an established simulation scoring checklist, a subjective quality rating, and whether the experts would recommend the scenario for use.

Seven experts with an average of 11 years of PEM training and an average of 10 years of simulation work scored simulations. While all 4 AI-generated cases had lower average scores on both quantitative assessments than their human-written counterparts, these results were not statistically significant. The majority of experts recommended each case to colleagues, regardless of source. There was moderate to excellent intrarater reliability.

Human-written cases received higher scores than the AI-written cases. While these scores were not statistically significant, these differences highlight the importance of keeping a human-expert influence when using AI to produce simulation cases. This ChatGPT tool can therefore be useful in assisting in the timely creation of scenarios, allowing facilitators, regardless of their technical skill with AI, to decrease cognitive load. Next steps should move beyond evaluating the scenario text to include evaluation of the experience of facilitating the ChatGPT-derived pediatric simulations as well as assessing the perspectives of simulation operators.