Setting performance standards for mannequin-based acute-care scenarios: an examinee-centered approach

Area of Science:

• Medical Education
• Assessment Methodology
• Simulation-Based Training

Background:

• Standard setting in medical assessments traditionally focuses on selected-response and performance-based tasks.
• Simulation-based training, particularly mannequin-based scenarios, presents unique challenges in evaluation due to complex, sequential actions.
• Reliable proficiency judgments for setting meaningful cut scores in mannequin-based evaluations are difficult to obtain.

Purpose of the Study:

• To explore the efficacy of an examinee-centered standard setting method for acute-care, mannequin-based scenarios.
• To determine if expert judgments, previously validated for standardized patient assessments, could establish defensible standards for mannequin simulations.

Main Methods:

• Nineteen physicians served as expert panelists to evaluate 12 simulation scenarios.
• Audio-video recordings of 8-10 performance samples per scenario, representing a range of trainee abilities, were reviewed.
• Panelists provided independent judgments based on a consensus definition of readiness for unsupervised practice to estimate scenario-specific cut scores.

Main Results:

• A moderately strong relationship between aggregate panelist judgments and performance scores allowed for meaningful standard estimation in 9 out of 12 scenarios.
• For 3 scenarios, panelist decision rules did not align with achievement measures.
• Estimated cut scores were consistent when scenarios were rated by independent split panels.

Conclusions:

• An examinee-centered method using aggregate expert judgments is suitable for standard setting in most acute-care, mannequin-based simulation scenarios.
• The success of this method relies on the availability of valid scenario scores and performance samples across the entire ability spectrum.