The Accuracy of Bayesian Model Fit Indices in Selecting Among Multidimensional Item Response Theory Models.

Area of Science:

• Psychometrics
• Statistical Modeling
• Educational Measurement

Background:

• Item Response Theory (IRT) models are crucial for analyzing rating scale data and determining dimensionality.
• Model comparisons based on predictive performance can be biased, favoring nested-dimensionality IRT models over non-nested ones.
• The extent of this bias, particularly with Bayesian estimation and specific comparison indices, remains unclear.

Purpose of the Study:

• To investigate the bias in Bayesian predictive performance indices when comparing nested- and non-nested-dimensionality IRT models.
• To assess the accuracy of four Bayesian indices in differentiating between these model types under various dimensional structures.
• To clarify the conditions under which nested-dimensionality models might be unfairly favored.

Main Methods:

• A simulation study was conducted to evaluate Bayesian predictive performance indices.
• Four indices were examined: Deviance Information Criterion (DIC), Pareto-smoothed importance sampling (PSIS-LOO), Watanabe-Akaike Information Criterion (WAIC), and log-predicted marginal likelihood (LPML).
• The study simulated data representing specific dimensional structures to test model differentiation accuracy.

Main Results:

• The Deviance Information Criterion (DIC) showed extreme bias, favoring nested-dimensionality models even when incorrect.
• Pareto-smoothed importance sampling (PSIS-LOO) demonstrated the least bias among the tested indices.
• Watanabe-Akaike Information Criterion (WAIC) and log-predicted marginal likelihood (LPML) also performed relatively well, with less bias than DIC.

Conclusions:

• Nested-dimensionality IRT models are not inherently favored when data represent specific dimensional structures if appropriate Bayesian predictive indices are used.
• The choice of model comparison index is critical for accurate dimensionality assessment in IRT.
• Pareto-smoothed importance sampling (PSIS-LOO) is recommended for reliable model comparison in these contexts.