Setting standards for performance tests: a pilot study of a three-level Angoff method.

Area of Science:

• Medical Education
• Assessment and Evaluation

Background:

• The Yes/No Angoff method is a standard-setting technique where judges assess borderline student performance on test items.
• Potential systematic severity or leniency in ratings can skew cut scores, especially when item success probability is 50%.

Purpose of the Study:

• To pilot a three-level Angoff method incorporating "Yes," "No," and "Maybe" categories for item ratings.
• To evaluate the impact of "Maybe" ratings on standard-setting in clinical skills exams.

Main Methods:

• Five judges applied the three-level Angoff method to seven checklists in a clinical skills exam for fourth-year students.
• Simulated rating severity by converting "Maybe" ratings to "Yes" or "No" to assess impact on cut scores.

Main Results:

• 41% of items received at least one "Maybe" rating, with 10% of all ratings falling into this category.
• Significant variation in case failure rates occurred under simulated severity conditions, though overall rates were not substantially affected.
• One judge contributed 83% of "Maybe" ratings; three judges did not use this category.

Conclusions:

• The three-level Angoff method maintains the simplicity of the Yes/No Angoff.
• It offers a way to manage items with midrange difficulty for borderline candidates, potentially improving standard-setting accuracy.