Related Experiment Video
Updated: Aug 6, 2026
Setting Up a Stroke Team Algorithm and Conducting Simulation-based Training in the Emergency Department - A Practical Guide
Published on: January 15, 2017
Validation of Checklist to Evaluate Emergency Medicine Competencies of Medical Students in Simulation Training
Nico Tannemann1, Mohamed Esmail1, Marius Gebauer2
1Department of Anesthesiology and Intensive Care Medicine, University Hospital Essen, University Duisburg-Essen, Essen, Germany.
Background:
Simulation-based training is crucial for acquiring emergency medicine competencies during medical education. In Germany, teaching is largely competency-based and guided by the National Competence Based Catalogue of Learning Objectives for Undergraduate Medical Education (NKLM) 2.0. Given the diversity of clinical scenarios and varying student expertise, a standardized assessment instrument is required for objective and reproducible performance evaluation. Moreover, such instruments are essential for educational research.
Methods:
We developed a checklist aligned with NKLM 2.0 and established emergency medicine guidelines. Content validity was ensured through expert panel review. The checklist assessed three domains (evaluation, background, and action) corresponding to key elements of the ISBAR framework. The instrument was applied to 265 medical students across different semesters and multiple simulation scenarios using purposive sampling and including all students from two semesters willing to participate. Construct validity was examined using factor analysis. Internal consistency was assessed using Cronbach's α, and interrater reliability by calculating variance attributable to rater and station effects.
Results:
After exclusion of one item with Kaiser-Meyer-Olkin estimator of 0.45, exploratory and confirmatory factor analyses supported a three-factor structure, with items loading on evaluation (CFA-loadings 0.831-0.901), background (0.582-0.736), and action (0.491-0.759). The CFA showed good model fit (CFI = 1.0, TLLI = 1.014, RMSEA < 0.001, SRMR = 0.045). Factors were moderately correlated (0.735-0.882), suggesting related dimensions of a common construct. Rater and station effects accounted for 14.2% of total variance. Cronbach's α was 0.730 overall, and 0.721 (evaluation), 0.557 (background), and 0.604 (action), respectively.
Conclusion:
Alignment with NKLM 2.0 supported content validity, while factor analysis confirmed the proposed construct structure. Overall internal consistency and acceptable rater-related variance as well as factor-correlation indicate that the checklist is a valid and reliable instrument for assessing emergency medicine competencies in undergraduate simulation training, though subscore interpretation remains limited due to low internal consistency of individual domains.
