Accounting for CAT-Induced Dependency in Differential Item Functioning Detection: A Multilevel Modeling Framework.

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Differential item functioning (DIF) detection is crucial for test fairness in computerized adaptive testing (CAT).
• Traditional DIF methods assume independent observations, which is violated in CAT due to adaptive item selection and provisional ability estimates.
• This structural dependency can lead to misleading results when applying standard DIF detection techniques.

Purpose of the Study:

• To propose and evaluate a novel two-level logistic model for enhanced DIF detection in CAT.
• To explicitly address the nuisance effects caused by CAT-induced structural dependency.
• To compare the performance of the proposed model against traditional single-level DIF methods.

Main Methods:

• Development of a two-level logistic regression model to account for dependencies in CAT.
• Comparison with single-level DIF models using a numeric example and extensive Monte Carlo simulations.
• Simulation conditions included varying test length, exposure control, ability estimator, DIF type, and DIF prevalence.

Main Results:

• The proposed two-level model demonstrated improved control of Type-I error (spurious DIF) compared to single-level models.
• The two-level model showed competitive statistical power, particularly under conditions of shorter test lengths and smaller item exposure rates.
• Model convergence varied systematically across simulation conditions, indicating an interplay between inferential accuracy and convergence reliability.

Conclusions:

• The two-level logistic model offers a promising approach for more accurate DIF detection in CAT.
• Future research should focus on jointly assessing model convergence and inferential performance in complex CAT DIF scenarios.
• This study highlights the importance of accounting for CAT-specific dependencies in DIF analysis.