Inferential Item-Fit Evaluation in Cognitive Diagnosis Modeling

Area of Science:

• Psychometrics
• Educational Measurement
• Cognitive Diagnosis Models

Background:

• Item-level fit evaluation is crucial for cognitive diagnosis models (CDMs).
• General CDMs require large sample sizes for reliable estimation and can suffer from poor attribute classification accuracy with small samples and low item quality.
• Balancing model fit and complexity is essential, adhering to the parsimony principle.

Purpose of the Study:

• To systematically examine the statistical properties of four inferential item-fit statistics: chi-squared (χ²), likelihood ratio (LR) test, Wald (W) test, and Lagrange multiplier (LM) test.
• To evaluate the performance of these statistics under various conditions, including sample size, correlational structure, test length, item quality, and generating model.

Main Methods:

• Monte Carlo simulations were employed to systematically manipulate key factors influencing statistical performance.
• The study examined Type I error rates and statistical power of the four item-fit statistics.
• Performance was assessed across different sample sizes, test lengths, item qualities, and data structures.

Main Results:

• The chi-squared (χ²) statistic demonstrated unacceptable statistical power.
• Likelihood ratio (LR) and Wald (W) tests generally outperformed the Lagrange multiplier (LM) test in terms of Type I error and power.
• All tested statistics were significantly affected by item quality, with acceptable performance primarily observed under high item quality conditions.

Conclusions:

• The effectiveness of common item-fit statistics in CDMs is highly contingent on item quality.
• While LR and W tests show better performance than LM, their utility is limited when item quality is low.
• Increasing sample size and test length can mitigate some performance issues, but assessing item fit in practical settings with low-quality items remains a significant challenge.