Objective standard setting (or truth in advertising).

Area of Science:

• Psychometrics
• Educational Measurement
• Assessment Policy

Background:

• Criterion referencing has been the predominant method for establishing passing standards in high-stakes examinations for four decades.
• Existing popular standard-setting models exhibit significant methodological weaknesses and limitations, extensively documented in prior research.

Purpose of the Study:

• To critically evaluate the limitations and methodological flaws inherent in widely used standard-setting models.
• To introduce and detail a novel approach, Objective Standard Setting (OSS), for establishing examination standards.
• To explore new considerations for validity within the context of standard-setting practices.

Main Methods:

• Review and critique of existing literature on popular standard-setting models.
• Detailed exposition of the Objective Standard Setting (OSS) methodology, developed in the early 1990s.
• Discussion of validity implications and psychometric expert responsibilities in standard setting.

Main Results:

• Identification of critical limitations and methodological weaknesses in prevalent criterion-referenced standard-setting approaches.
• Successful application of the Objective Standard Setting (OSS) model across diverse settings since its inception.
• The paper provides the first published account of the OSS methodology.

Conclusions:

• Popular standard-setting methods, despite good intentions, require closer scrutiny of their underlying psychometric principles.
• Psychometric experts must possess a comprehensive understanding of standard-setting models, their validity, and their implications.
• The Objective Standard Setting (OSS) model offers a robust alternative for establishing defensible examination standards.