Evidence for the validity of a Rasch model technique for identifying differential item functioning

Area of Science:

• Medical Education
• Psychometrics
• Health Professions Education

Background:

• Differential Item Functioning (DIF) analysis is crucial for ensuring fairness in high-stakes examinations.
• Interpreting DIF results can be challenging, particularly when comparing diverse demographic groups.

Purpose of the Study:

• To analyze DIF in a medical specialty certification examination.
• To evaluate the interpretability of DIF results using a simple Rasch model procedure.
• To assess the utility of understanding group differences for DIF interpretation.

Main Methods:

• Employed a simple Rasch model procedure for DIF analysis.
• Compared physicians from different subspecialties.
• Compared physicians qualifying through different experiential pathways.

Main Results:

• DIF analyses yielded readily interpretable results.
• Interpretability was linked to known differences between the compared physician groups.
• The Rasch model procedure provided validity evidence for DIF evaluation.

Conclusions:

• Complex DIF procedures are unnecessary for interpretable results when group differences are known.
• Difficulty in interpreting DIF for demographic groups may stem from a lack of pertinent knowledge, not methodological limitations.