Effect of Differential Item Functioning on Computer Adaptive Testing Under Different Conditions.

Area of Science:

• Psychometrics
• Educational Measurement
• Computerized Adaptive Testing

Background:

• Differential Item Functioning (DIF) can compromise the validity of test scores.
• Computer Adaptive Tests (CATs) are susceptible to the effects of DIF due to their dynamic item selection.
• Understanding DIF's impact is crucial for maintaining test precision and effectiveness.

Purpose of the Study:

• To investigate the effects of DIF on measurement precision, test information function (TIF), and overall test effectiveness in CATs.
• To analyze how varying DIF parameters influence key psychometric properties within a CAT framework.
• To quantify the impact of DIF on the accuracy and reliability of ability estimates.

Main Methods:

• Simulated data generation using Rstudio, manipulating item pool size, DIF type, and DIF percentage.
• CAT simulations incorporating various item selection and test termination rules.
• Calculation of measurement precision, TIF, and test effectiveness under different DIF conditions.

Main Results:

• Differential item functioning (DIF) was found to negatively affect measurement precision, TIF, and test effectiveness.
• The percentage of DIF items and the type of DIF demonstrated statistically significant negative impacts on measurement precision.
• CATs with higher percentages of DIF items exhibited reduced accuracy in ability estimation.

Conclusions:

• DIF poses a significant threat to the psychometric integrity of CATs.
• Test developers must carefully consider and mitigate DIF to ensure accurate and reliable measurement.
• Further research into DIF detection and control methods within CATs is warranted.