A Comparison of Approaches for Mastery Learning Standard Setting

Area of Science:

• Medical Education
• Procedural Skills Assessment
• Patient Safety

Background:

• Establishing defensible minimum passing standards (MPSs) is crucial for evaluating learner performance in health professions education.
• Traditional methods like Angoff and Hofstee are commonly used for standard-setting in procedural skills examinations.
• Simulation-based mastery learning (SBML) is an effective educational approach for developing procedural competency.

Purpose of the Study:

• To compare the minimum passing standards (MPSs) derived from traditional Angoff and Hofstee methods with those from Mastery Angoff and Patient-Safety approaches for central venous catheter (CVC) insertion skills.
• To evaluate the impact of these different MPSs on the historical performance data of residents trained in a CVC insertion SBML curriculum.

Main Methods:

• Twelve CVC insertion experts established MPSs for internal jugular (IJ) and subclavian (SC) CVC insertion checklists in April-May 2015.
• Four standard-setting methods were employed: Angoff, Hofstee, Mastery Angoff, and Patient-Safety.
• The established MPSs were applied to historical performance data of 143 internal medicine and emergency medicine residents who completed a CVC insertion SBML curriculum.

Main Results:

• The Mastery Angoff and Patient-Safety methods yielded higher MPSs (98% for both IJ and SC) compared to Angoff (IJ 91%, SC 90%) and Hofstee (IJ 88%, SC 90%).
• Applying the higher 98% MPS would have required an additional 45% of residents (55/123) to practice and retest for IJ insertion and 28% (36/130) for SC insertion, who had previously passed using lower MPSs.
• This indicates a significant difference in pass rates depending on the standard-setting methodology used.

Conclusions:

• More rigorous standard-setting approaches, specifically Mastery Angoff and Patient-Safety, result in higher minimum passing standards for CVC insertion skills within an SBML framework.
• These findings suggest that traditional methods may underestimate the level of proficiency required for safe CVC insertion.
• Further research is warranted to investigate the correlation between these higher MPSs and actual patient outcomes.