Setting and maintaining standards in multiple choice examinations: AMEE Guide No. 37.

Area of Science:

• Educational Measurement
• Psychometrics
• Assessment and Evaluation

Background:

• Standard setting in examinations involves subjective judgment, leading to potential errors like false positives and negatives.
• Maintaining a fixed pass/fail score is inappropriate due to varying examination difficulty.

Purpose of the Study:

• To minimize errors in standard setting by accounting for examination difficulty variations.
• To compare norm-referenced and criterion-referenced standard-setting approaches.
• To provide guidance on selecting, training, and instructing judges in the standard-setting process.

Main Methods:

• Review and illustration of commonly used standard-setting methods.
• Discussion of advantages and disadvantages of each method.
• Explanation of test equating for maintaining standards across examinations.
• Proposal of a practical method for pre-determined standard setting using item difficulty.

Main Results:

• Criterion-referenced standards are preferred for certifying individual competence.
• Common errors in standard setting by judges are identified.
• A method for equating tests to ensure consistent standards over time is presented.

Conclusions:

• Standard setting must account for test difficulty and minimize judgment errors.
• Proper selection and training of judges are crucial for reliable standard setting.
• A practical approach to establishing pre-determined standards based on item difficulty is proposed.