Standard-setting in written and interactive (oral) specialty certification examinations: issues, models, methods,...

Area of Science:

• Medical Education
• Psychometrics
• Health Professions Education

Background:

• Standard-setting is crucial for valid and reliable health professions examinations.
• Current methods for written and interactive (oral) examinations have limitations.
• Understanding the psychological and psychometric aspects of standard-setting is essential.

Purpose of the Study:

• To classify and evaluate existing standard-setting methods for health professions exams.
• To propose a framework for understanding standard-setting as an interactive system.
• To provide design considerations for systematizing judgments in interactive examinations.

Main Methods:

• Review and classification of current standard-setting methodologies.
• Analysis of strengths and weaknesses of normative and content-referenced methods for written exams.
• Discussion of judgment-based standards for interactive examinations.

Main Results:

• Standard-setting is a complex interactive system involving decision-makers, subject area, and methods.
• Both psychological/social psychological and psychometric factors influence standard-setting.
• Normative and content-referenced methods have distinct applications and limitations for written exams.

Conclusions:

• Standard-setting relies on a foundation of judgment, influenced by multiple interacting factors.
• Absolute standards are inherent in interactive examinations, requiring careful design.
• Systematic approaches are needed to enhance the reliability and validity of health professions examinations.