Related Experiment Video
Updated: Jul 13, 2025
00:08
A Cross-Disciplinary and Multi-Modal Experimental Design for Studying Near-Real-Time Authentic Examination Experiences
Published on: September 4, 2019
7.1K
Towards a more nuanced conceptualisation of differential examiner stringency in OSCEs
1School of Medicine, University of Leeds, Leeds, LS2 JT, UK. m.s.homer@leeds.ac.uk.
Advances in Health Sciences Education : Theory and Practice
|October 16, 2023
Summary
Examiner stringency in Objective Structured Clinical Examinations (OSCEs) impacts scoring validity. This study reveals examiners affect scores through both consistent bias and differential grading, influencing pass/fail decisions.
Area of Science:
- Medical education research
- Assessment and evaluation in health professions education
- Psychometrics in performance-based assessment
Background:
- Systematic differences in examiner scoring (stringency) in Objective Structured Clinical Examinations (OSCEs) can compromise outcome validity.
- Traditional analyses often overlook global grades, focusing solely on checklist or domain scores.
- A nuanced understanding of examiner variability is crucial for reliable assessment.
Purpose of the Study:
- To develop a more sophisticated model of examiner stringency in OSCEs using a large candidate-level dataset.
- To investigate how examiners vary not only in their overall stringency but also in their discrimination across different grade levels.
- To quantify the impact of examiner variability on candidate pass/fail decisions.
Main Methods:
- Analysis of a large candidate-level OSCE dataset.
- Modeling station scores incorporating candidate ability, station difficulty, examiner stringency, and examiner grade discrimination.
- Quantification of variance components attributed to examiners, candidates, and stations.
Main Results:
- Examiners contribute significantly to score variance through both consistent stringency (34%) and differential grade discrimination (7%).
- Candidate and station factors account for minimal score variance (3% and 2% respectively) after accounting for examiner effects.
- Differential examiner stringency leads to approximately 5% false positive and 5% false negative rates at the station level, reducing to 0.4% and 3.3% at the exam level.
Conclusions:
- Examiners exhibit variability in judging candidates in qualitatively distinct ways, impacting assessment reliability.
- Institutions must ensure broad sampling of examiners across sufficient stations to defend OSCE-level decisions.
- Examiner training should address global grading and its combined effects with scoring on candidate outcomes.
Related Concept Videos
01:24Stress Concentrations
336
Stress concentration is when stress intensifies near discontinuities such as holes or abrupt cross-sectional changes in a structural member. This localized stress can often surpass the average stress within the member. The stress distribution in flat bars, either with a circular hole or varying widths connected by fillets, can be determined experimentally using a photoelastic method. The results are based on ratios of geometric parameters like the ratio of the hole's radius to the smaller...
336
01:25Yield Criteria for Ductile Materials under Plane Stress
168
In designing structural elements and machine parts using ductile materials, it is crucial to ensure that these components withstand applied stresses without yielding. Yielding is initially determined through a tensile test, which evaluates the material's response to uniaxial stress. However, tensile stress is insufficient when components face biaxial or plane stress conditions This condition requires advanced criteria to predict failure.
The Maximum Shearing Stress Criterion, also known as...
The Maximum Shearing Stress Criterion, also known as...
168
01:05Difference from Background: Limit of Detection
6.4K
The limit of detection (LOD) is the smallest amount of analyte that can be distinguished from the background noise. The LOD value corresponds to the concentration at which the analyte signal is three times larger than the standard deviation of the blank signal. Below this value, the analyte signal cannot be differentiated from the background noise. It is calculated by dividing the calibration slope by 3 times the standard deviation of the blank signals.
The LOD indicates the presence or absence...
The LOD indicates the presence or absence...
6.4K