Exploring the Use of a Large Language Model in Simulation Debriefing: An Observational Simulation-Based Pilot Study.

Area of Science:

• Medical Simulation
• Artificial Intelligence in Education
• Human-Computer Interaction

Background:

• Simulation-based education requires effective debriefing, which is a complex task with high facilitator workload.
• Generative artificial intelligence (AI) presents an opportunity to support simulation facilitators and enhance debriefing quality.
• Large language models (LLMs) can potentially decrease facilitator task load and enable more comprehensive debriefs.

Purpose of the Study:

• To explore the use of a large language model (LLM) for facilitating simulation debriefings.
• To assess the impact of LLM integration on facilitator task load and simulation debriefing quality.
• To evaluate learner and facilitator perceptions of AI technologies in simulation-based education.

Main Methods:

• A prospective, observational, simulation-based pilot study at Yale University School of Medicine.
• Real-time simulation transcriptions were input into the GPT-4o LLM to generate debriefing scripts.
• Facilitator and learner task workload (NASA-TLX) and perceptions of AI were assessed via surveys.

Main Results:

• Four facilitators and 25 learners participated; all data were self-reported.
• Participants showed strong enthusiasm for AI integration (mean Likert scores 4.75/5 for facilitators, 4.0/5 for learners).
• AI was perceived to help maintain focus (4.8/5), support learning objectives (4.2/5), and minimize distractions (4.6/5). Facilitators and learners reported moderate to high mental demand (NASA-TLX scores 10.8/21 and 9.9/21).

Conclusions:

• LLM integration aids debriefing by organizing complex simulation information.
• LLMs show potential to enhance simulation-based debrief quality, despite reported facilitator task load.
• Continuous human oversight is essential when implementing AI in simulation debriefings.