Related Experiment Video
Updated: Jul 20, 2026
05:21
Computerized Adaptive Testing System of Functional Assessment of Stroke
Published on: January 7, 2019
Assessment of examiner leniency and stringency ('hawk-dove effect') in the MRCP(UK) clinical examination (PACES)
I C McManus1, M Thompson, J Mollon
1Department of Psychology, University College London, Gower Street, London WC1E 6BT, UK. i.mcmanus@ucl.ac.uk
BMC Medical Education
|August 22, 2006
Summary
Examiner stringency in clinical exams varies, impacting candidate scores. Rasch modelling quantifies this variation, enabling fairer assessments through adjusted marks or examiner pairing.
Area of Science:
- Medical Education
- Psychometrics
- Statistical Modeling
Background:
- The hawk-dove problem highlights variability in examiner stringency in clinical examinations.
- Previous studies have lacked statistical estimation of this effect in large-scale assessments.
- This study addresses the need for quantitative analysis of examiner leniency-stringency.
Purpose of the Study:
- To statistically estimate the effect of examiner stringency in the MRCP(UK) clinical examination (PACES).
- To identify factors associated with examiner leniency-stringency.
- To evaluate methods for accounting for examiner stringency in scoring.
Main Methods:
- Utilized FACETS software for multi-facet Rasch modeling of paired examiner judgments.
- Analyzed data from 10,145 candidates and 1,259 examiners across nine PACES diets (2001-2004).
- Examined 142,030 total marks, correlating examiner demographics with stringency.
Main Results:
- Examiner differences accounted for 12% of the main effect variance, compared to 87% for candidate differences.
- Greater examiner experience and minority ethnicity were associated with increased stringency.
- Score adjustments minimally impacted overall pass/fail outcomes (95.9% unchanged).
Conclusions:
- Examiner leniency-stringency is a quantifiable effect in clinical examinations, estimable via Rasch modeling.
- Demographic factors show some correlation with stringency, which appears consistent over time.
- Adjusting scores or pairing examiners of differing stringency can mitigate bias and ensure fairer assessments.
