Exploring the Evidence to Interpret Differential Item Functioning via Response Process Data.

Area of Science:

• Educational Measurement
• Psychometrics
• Data Science

Background:

• Differential item functioning (DIF) evaluation is crucial for ensuring assessment fairness across diverse subgroups.
• Traditional DIF methods relying solely on item response scores present interpretability challenges.
• Response process data offer novel insights into examinee behaviors, aiding DIF interpretation.

Purpose of the Study:

• To explore the utility of response process data features for enhancing the interpretability of DIF items.
• To focus on gender-based DIF within the Programme for International Assessment of Adult Competencies (PIAAC) 2012 numeracy assessment.
• To identify key process data features that explain DIF.

Main Methods:

• Utilized random forest and logistic regression with ridge regularization.
• Investigated associations between process data features and DIF items.
• Assessed model performance across varying percentages of DIF items to simulate real-world conditions.

Main Results:

• Timing and action-sequence features were found to be highly informative.
• These features effectively revealed response process differences between gender groups.
• The combination of process data features significantly enhanced DIF item interpretability.

Conclusions:

• Response process data provide a feasible method for understanding and interpreting DIF items.
• This approach can illuminate reasons for discrepancies between DIF statistics and expert reviews.
• Leveraging process data can help identify and mitigate irrelevant factors affecting measurement equity.