Investigating the Effects of Differential Item Functioning on Proficiency Classification.

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Differential item functioning (DIF) is a critical concern in standardized testing, potentially biasing proficiency classification.
• Understanding DIF's impact on classification accuracy across different demographic groups is essential for test fairness.

Purpose of the Study:

• To comprehensively evaluate how differential item functioning (DIF) affects proficiency classification outcomes.
• To investigate the impact of varying item- and test-level DIF magnitudes on classification accuracy at multiple decision points.

Main Methods:

• Monte Carlo simulation was employed to systematically vary DIF magnitudes.
• The study analyzed the effects of DIF on classification error rates for reference and focal groups.

Main Results:

• DIF influences proficiency classification by creating differential error rates between groups, rather than reducing overall correct classification.
• Multiple items exhibiting low levels of DIF can be as detrimental as high-level DIF items but are harder to detect.
• The study details how DIF impacts classification errors across various cut scores.

Conclusions:

• The presence of DIF can lead to inequitable proficiency classifications, disproportionately affecting certain groups.
• Current DIF and differential bundle functioning (DBF) detection methods may not adequately identify cumulative effects of low-level DIF.
• Further research into robust detection methods for subtle DIF effects is warranted to ensure fair and accurate proficiency classification.