Assessing Item-Level Fit for Higher Order Item Response Theory Models

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Item-level fit testing is crucial for scale development, guiding item revision or deletion.
• Existing item-level fit indices are not directly applicable to higher-order item response theory (HO-IRT) models.
• Higher-order IRT models are widely used but lack appropriate fit assessment tools.

Purpose of the Study:

• To extend chi-square-based item-level fit indices to HO-IRT models.
• To evaluate the performance of these extended fit indices using simulation studies.
• To identify the most effective fit index for detecting misfit in HO-IRT models.

Main Methods:

• Chi-square-based fit indices (Yen's Q1, McKinley and Mill's G2, Orlando and Thissen's S-X^2, S-G^2) were adapted for HO-IRT models.
• Extensive simulation studies were conducted to assess the performance of the extended indices.
• Factors manipulated included test structure, sample size, dimensionality, and proportion of misfitting items.

Main Results:

• The simulation results indicated that the S-G^2 fit index performed promisingly for HO-IRT models.
• The study analyzed false positive rates and correct detection rates across various conditions.
• The effectiveness of S-G^2 was evaluated under different sources of misfit, including response function deviations and structural misspecification.

Conclusions:

• The S-G^2 fit index is a viable and promising tool for assessing item-level fit within HO-IRT frameworks.
• This extension facilitates more rigorous scale development and model evaluation in complex measurement contexts.
• The findings provide practical guidance for researchers using HO-IRT models in psychometric analysis.