Secure AI-assisted angoff standard-setting for single best answer questions: A non-inferiority validation study

The Angoff method sets exam pass scores using expert estimates of the probability that a borderline candidate will answer each question correctly; this process is time-intensive and variable. We evaluated whether AI-derived Angoff estimates are non-inferior to human judgements for single best answer (SBA) questions using a secure offline question feature-extraction tool.

We conducted a methodological study in a UK Physician Associate programme. A faculty survey informed a borderline Year-2 student descriptor. ExamFeats, a secure offline tool, extracted surface features from 1,003 historical and 100 new SBAs. Three AI models were evaluated: (A) a large language model (LLM) using extracted features and the borderline-student descriptor; (B) a machine-learning model (ML) using ridge regression and (C) a hybrid combining LLM and ML predictions. Human Angoff ratings for the 100 new items were averaged across 3-4 faculty members. The primary outcome was the mean difference between AI and human Angoff scores, assessed against a prespecified 10% non-inferiority margin, with additional analysis of discordant items (>10% difference).

For the 100 new SBAs, human Angoff estimates and all AI models produced closely aligned distributions, with minimal systematic differences. The human mean Angoff score was 60.3%, while Model A, Model B and Model C produced mean estimates of 60.8%, 60.0% and 60.3%, respectively. There were no significant differences between models (p = 0.41) and 95% confidence-intervals lay within the non-inferiority margin. Discordance between human and model prediction occurred in 33% of SBAs but this was not explained by differences in extracted surface item features.

AI-derived Angoff estimates reproduced panel-level Angoff behaviour within the prespecified error bounds and could reduce the time and cognitive burden of standard setting. The secure feature-extraction tool enabled AI-assisted standard setting without sharing item text. However, question-level discordance indicates that AI should augment rather than replace expert panels.