Variability in Testing Scenario Complexity Reduces the Validity of the Advanced Trauma Life Support Certification...

The Advanced Trauma Life Support (ATLS) course plays a pivotal role in the trauma education of healthcare professionals on a global scale. To achieve certification, participants are evaluated in simulated trauma scenarios. The McGill Simulation Complexity Score (MSCS) is a trauma-specific tool with supporting validity evidence, developed to quantify the complexity of trauma simulations. This study aims to assess the heterogeneity and complexity of the trauma scenarios used to assess ATLS candidates.

Four independent reviewers utilized the MSCS to assess the 21 ATLS assessment scenarios for complexity. Each resuscitation component (Airway, Breathing, Circulation, Disability, Exposure) was given a score from 0 (lowest complexity) to 4 (highest complexity), with a total case score between 0 and 20. Variability between case complexity and inter-reviewer reliability was assessed.

The mean complexity score was 7.7 (standard deviation 0.61), with scores ranging from 3.5 to 13.25. Inter-rater reliability was high with an interclass correlation coefficient of 0.80. Of the individual resuscitation elements scored, the circulation component demonstrated the highest average complexity while the exposure component contributed the least complexity with mean scores of 2.38 and 0.82, respectively.

There is variability in the complexity scores of the ATLS assessment simulation scenarios. This variability is a threat to the validity of the ATLS course assessment. Given the requirement for ATLS certification in many health professions training programs and employment opportunities, reform of course assessment scenarios should be considered to standardize complexity and support assessment validity.