Differential Item Functioning Detection Across Two Methods of Defining Group Comparisons: Pairwise and Composite...

Area of Science:

• Educational Measurement
• Psychometrics
• Statistics

Background:

• Differential Item Functioning (DIF) is crucial for ensuring fairness in standardized testing.
• The definition of comparison groups significantly influences DIF detection outcomes.
• Existing research lacks a clear consensus on optimal group definition strategies for DIF analysis.

Purpose of the Study:

• To compare two methods for defining groups in DIF detection: pairwise and composite comparisons.
• To investigate how the choice of group definition reflects different conceptualizations of fairness in DIF studies.
• To provide empirical evidence supporting the use of specific group definition approaches.

Main Methods:

• A simulation study using data from a 60-item ACT Mathematics test.
• Application of the unsigned area measure method (Raju) for DIF detection.
• Comparison of Type I error rates and false discovery rates between pairwise and composite group methods.

Main Results:

• The amount and interpretation of flagged DIF varied significantly between pairwise and composite group methods.
• Composite group approaches demonstrated potential benefits in DIF detection.
• Observed Type I error rates and false discovery rates differed across the two group definition strategies.

Conclusions:

• The choice between pairwise and composite group definitions in DIF analysis is intrinsically linked to the definition of fairness.
• Composite group comparisons may offer advantages for more robust DIF detection.
• Recommendations for practical application in educational measurement are provided based on the findings.