Comparison of Relative Fit Indices for Diagnostic Model Selection

The purpose of this study was to thoroughly examine the performance of three information-based fit indices-Akaike's Information Criterion (AIC), Bayesian Information Criterion (BIC), and sample-size-adjusted BIC (SABIC)-using the log-linear cognitive diagnosis model and a set of well-known item response theory (IRT) models. Two simulation studies were conducted to examine the extent to which relative fit indices can identify the generating model under a variety of data conditions and model misspecifications. Generally, indices performed better when item quality was stronger. When the IRT was the generating model, all three indices correctly selected the IRT model for all replications. When the true model was a diagnostic classification model, for all three fit indices, the multidimensional IRT model was incorrectly selected as frequently as 70% of the replications. The results of this study identify situations for researchers where commonly used-and typically well-performing-fit indices may not be appropriate to compare models for selection.