Determining intra-standard-setter inconsistency in the Angoff method using the three-parameter item response theory

Area of Science:

• Medical Education
• Psychometrics
• Educational Assessment

Background:

• Standard-setting in medical education is crucial for determining pass marks.
• The Angoff method is widely used but can exhibit intra-standard-setter variability.
• Latent trait theory provides a framework for understanding assessment variability.

Discussion:

• This study applied the three-parameter item response theory (IRT) to analyze Angoff ratings from 358 medical students.
• The analysis focused on measuring intra-standard-setter variability and inter-standard-setter validity.
• Data were collected from two summative knowledge-based assessments.

Key Insights:

• The three-parameter IRT model effectively analyzes individual standard-setter results.
• Standard-setters demonstrated considerable consistency across both assessments.
• A strong positive correlation was observed between mean Angoff ratings and conditional probability, indicating inter-standard-setter validity.

Outlook:

• Adopting this IRT-based methodology can identify and minimize judgmental inconsistencies among standard-setters.
• This approach aids in reducing false positive and false negative decisions in educational assessments.
• Further research could explore the application of this methodology in diverse assessment contexts.