Testing Differential Item Functioning in Small Samples.

Area of Science:

• Psychometrics
• Statistical modeling
• Social and behavioral sciences

Background:

• Differential item functioning (DIF) can obscure true group differences in latent constructs.
• Item response theory (IRT) methods for DIF testing often assume large sample sizes for stable parameter estimation.
• Small sample sizes are common in social and behavioral research, yet DIF methods for these conditions are less studied.

Purpose of the Study:

• To investigate how model complexity impacts the detection of DIF in small sample sizes.
• To compare the performance of logistic regression, 1-parameter logistic IRT, and 2-parameter logistic IRT models in small samples for DIF analysis.

Main Methods:

• A simulation study was conducted using data generated from a 2-parameter logistic IRT model.
• Three models of varying complexity were compared: logistic regression with sum scores, 1-parameter logistic IRT, and 2-parameter logistic IRT.
• An empirical example involving adolescent substance use data was analyzed.

Main Results:

• Parsimonious models (logistic regression and 1-parameter IRT) demonstrated more powerful DIF detection in small samples.
• Simpler models adequately controlled for Type I error, even when data followed a more complex 2-parameter IRT model.
• Evidence was provided regarding minimum sample sizes for DIF detection and the advisability of multiple testing corrections.

Conclusions:

• In small samples, utilizing less complex models for DIF analysis enhances statistical power while maintaining acceptable error rates.
• Researchers are advised to consider model parsimony when conducting DIF analyses with limited data.
• Recommendations are provided for applied researchers facing DIF analysis challenges in small sample contexts.