A Propensity Score Method for Investigating Differential Item Functioning in Performance Assessment.

Area of Science:

• Educational Measurement
• Psychometrics
• Quantitative Psychology

Background:

• Performance assessment data present unique analytical challenges, including continuous scores and the absence of reliable internal ability proxies.
• Traditional differential item functioning (DIF) methods may struggle with these data characteristics, potentially leading to biased results.

Purpose of the Study:

• To introduce and validate a novel differential item functioning (DIF) method specifically designed for performance assessment data.
• To address the challenges posed by continuous task scores and the lack of a direct measure of ability or aptitude.

Main Methods:

• The proposed method integrates propensity score matching with linear mixed effects models.
• Propensity score matching is employed in the first stage to create statistically equivalent groups by controlling for pre-existing differences.
• Linear mixed effects models are then used in the second stage to conduct DIF analysis on the matched data.

Main Results:

• The propensity DIF method was successfully demonstrated using a high-stakes functional English language proficiency test.
• The study investigated DIF related to education in a writing component featuring continuously scored performance tasks.
• The method proved effective in addressing group differences and analyzing DIF in complex performance assessments.

Conclusions:

• The novel propensity score matching-based DIF method offers a robust approach for analyzing performance assessment data.
• This method effectively handles continuous scores and the absence of internal ability proxies.
• The approach is applicable beyond language testing to various performance assessment contexts.