Secure AI-assisted angoff standard-setting for single best answer questions: A non-inferiority validation study

Area of Science:

• Medical Education
• Artificial Intelligence in Assessment

Background:

• The Angoff method for setting exam pass scores is expert-driven, time-consuming, and prone to variability.
• Evaluating AI-driven Angoff estimates offers a potential solution to improve efficiency and consistency.

Purpose of the Study:

• To assess if AI-derived Angoff estimates are non-inferior to human judgments for single best answer (SBA) questions.
• To explore the utility of a secure, offline AI tool for feature extraction in standard-setting.

Main Methods:

• A methodological study in a UK Physician Associate program utilized a borderline student descriptor and an AI tool (ExamFeats) to extract features from SBAs.
• Three AI models (LLM, ML, Hybrid) were compared against average human Angoff ratings for 100 new SBAs, with a 10% non-inferiority margin.

Main Results:

• AI models produced mean Angoff estimates closely aligned with human scores (60.0%-60.8% vs. 60.3% human).
• No significant differences were found between AI models and human ratings, with confidence intervals within the non-inferiority margin.
• Discordance occurred in 33% of SBAs, not explained by item features, suggesting AI should augment, not replace, human expertise.

Conclusions:

• AI-derived Angoff estimates can replicate panel-level behavior within acceptable error bounds, potentially reducing the burden of standard setting.
• A secure feature-extraction tool allows AI-assisted standard setting without compromising item security.
• Question-level discordance highlights the need for AI to supplement, rather than substitute, expert judgment in exam standard setting.