A Novel Application of the Modified Angoff Method to Rate Case Difficulty in Simulation-Based Research

Area of Science:

• Medical Education
• Simulation Research
• Physiological Stress Measurement

Background:

• Controlling for scenario difficulty in repeated measures simulation research is challenging.
• The Modified Angoff Method was assessed for its feasibility in achieving expert consensus on medical simulation difficulty.
• Physiological stress responses of participants were compared with expert difficulty ratings.

Purpose of the Study:

• To evaluate the feasibility of the Modified Angoff Method for establishing expert consensus on medical simulation difficulty.
• To compare expert difficulty ratings with participant physiological stress levels (heart rate variability).

Main Methods:

• Emergency medicine physicians used the Modified Angoff Method to estimate the percentage of residents performing critical actions for 8 scenarios.
• Consensus was defined as a standard deviation (SD) of less than 10% in expert estimations.
• Twenty-five residents performed 6 consensus-achieved scenarios, with heart rate variability (HRV) measured.

Main Results:

• Expert consensus (SD < 10%) was reached for 6/8 scenarios after two rounds of ratings.
• High inter-rater reliability was observed (ICC = 0.84 for PGY3, 0.89 for PGY4).
• No significant difference in HRV change was found between teams or scenarios; Modified Angoff scores did not predict HRV.

Conclusions:

• The Modified Angoff Method is a feasible tool for achieving expert consensus on medical simulation difficulty.
• While simulations induced physiological stress, it was not correlated with expert difficulty ratings.
• This method may streamline simulation piloting, reducing time and resources for educational and research applications.