A standard setting method with the best performing students as point of reference: practical and affordable

Area of Science:

• Educational Measurement
• Psychometrics

Background:

• Teachers often prefer criterion-referenced standard setting.
• Expert panels are costly, leading to use of pre-fixed percentages or norm-referenced methods.
• Existing methods present significant disadvantages in practice.

Purpose of the Study:

• To evaluate the advantages and disadvantages of common criterion-referenced and norm-referenced standard setting methods.
• To introduce and assess a novel compromise method for standard setting.

Main Methods:

• Analysis of historical data from 54 norm-referenced (Maastricht) and 52 criterion-referenced (Groningen) tests.
• Application of a compromise model (Cohen's method) to the Groningen tests.

Main Results:

• Norm-referenced methods showed large cut-off score variations (15-46%) but stable failure rates (~17%).
• Criterion-referenced methods with a 60% cut-off exhibited high failure rate variability (17-97%).
• The compromise method effectively reduced variations in both cut-off scores and failure rates.

Conclusions:

• Both traditional criterion-referenced and norm-referenced standard setting methods have inherent disadvantages.
• The proposed compromise method offers a more acceptable solution by mitigating the drawbacks of existing approaches.
• This novel method is a cost-effective alternative to panel-based standard setting.