Standard setting in Australian medical schools

Area of Science:

• Medical Education
• Assessment and Evaluation
• Quality Assurance

Background:

• Standard setting is crucial for medical program quality assurance.
• This study examined standard-setting methods across 13 Australian medical schools in 2014.
• The Australian Medical Schools Assessment Collaboration (AMSAC) provided benchmarking data.

Purpose of the Study:

• To identify and compare the impact of standard-setting methods used by Australian medical schools.
• To analyze the variation in implementation of common techniques like Modified Angoff.
• To assess the influence of standard-setting on student performance evaluation.

Main Methods:

• A survey identified standard-setting procedures used by participating schools.
• Standard-setting data for 49 multiple-choice items were collated.
• Analyses focused on nine schools, examining methods and panel member characteristics.

Main Results:

• Significant differences were found in standard-setting methods and their implementation.
• The correlation of item standard settings varied widely between schools (-0.116 to 0.632).
• Panel members tended to misjudge item difficulty; a median cut-score of 55% was observed. High curriculum engagement correlated with lower borderline candidate expectations (p=0.044).

Conclusions:

• Australian medical schools employ diverse standard-setting techniques, with varied implementation.
• The Modified Angoff method, though common, showed marked variation in application.
• Standard-setting methods influenced performance distribution, causing up to a 10% variation in passing standards, highlighting a need for consistency.