Identifying and Addressing Item Bias in the General Pediatrics Certifying Examination

Area of Science:

• Medical Education
• Psychometrics
• Health Equity

Background:

• Medical certification examinations must be free from bias to ensure equitable assessment of physician competency.
• The American Board of Pediatrics (ABP) implemented a rigorous process to evaluate potential bias in its General Pediatrics Certifying Examination.
• Ensuring fairness in high-stakes examinations is crucial for maintaining public trust and professional standards.

Purpose of the Study:

• To describe a two-phase process for identifying and removing item-level bias based on gender, race, and ethnicity.
• To assess the effectiveness of differential item functioning (DIF) analysis and expert review in detecting biased examination items.
• To improve the quality and fairness of future pediatric certification examinations.

Main Methods:

• Phase 1: Differential item functioning (DIF) analysis was employed to statistically identify examination items where performance differed between demographic subgroups (gender, race/ethnicity) after controlling for overall knowledge.
• Phase 2: A Bias and Sensitivity Review (BSR) panel, composed of diverse subject matter experts, reviewed items flagged by DIF analysis to identify potential sources of bias.
• The BSR panel assessed items for biased language or characteristics that could explain performance disparities.

Main Results:

• No items were flagged for DIF based on gender in the 2021 examination.
• 2.8% of examination items were flagged for DIF based on race and ethnicity.
• Of the items flagged for race/ethnicity, 14.3% (0.4% of total items) were identified by the BSR panel as containing biased language and were recommended for removal.

Conclusions:

• The implemented two-phase process effectively identified and addressed potential bias in the General Pediatrics Certifying Examination.
• Statistical DIF analysis coupled with expert review provides a robust method for detecting and mitigating item bias.
• Ongoing application of this process aims to enhance understanding of item performance and refine item development guidelines for greater equity.