Using differential item functioning to evaluate potential bias in a high stakes postgraduate knowledge based...

Area of Science:

• Medical Education
• Psychometrics
• Health Professions Education

Background:

• Fairness in assessment is crucial for defensible evaluations, ensuring candidates perform based on ability, not background.
• Performance disparities across demographic groups (ethnicity, sex) are observed in many assessments.
• Routine analysis of examinations, including Differential Item Functioning (DIF), is necessary to identify and address potential progression barriers for specific groups.

Purpose of the Study:

• To investigate assessment fairness using DIF for an international postgraduate examination in internal medicine.
• To identify specific questions exhibiting bias between ethnically white and non-white UK graduates, and between male and female UK graduates.

Main Methods:

• Differential Item Functioning (DIF) analysis was applied to 2773 questions across 14 sittings of a summative postgraduate examination.
• The study included 13,694 candidates, comparing ethnically white UK graduates with ethnically non-white UK graduates, and male UK graduates with female UK graduates.

Main Results:

• Only eight out of 2773 questions (0.29%) showed notable DIF after correction for multiple comparisons.
• A panel of blinded clinician assessors found no plausible explanations for the DIF in these identified questions.
• The identified questions did not significantly impact overall cohort performance, and group performance differences were not explained by these unfair questions.

Conclusions:

• Differential Item Functioning (DIF) is a valuable tool for exploring assessment fairness at the question level, particularly in high-stakes examinations.
• The study found very few questions with notable DIF, indicating that group-level performance differences in this examination are not attributable to unfairness in individual assessment items.
• The findings suggest that broader factors beyond individual question bias likely contribute to observed disparities in passing rates between demographic groups.