Assessing Item-Level Fit for the DINA Model.

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Diagnostic classification models (DCMs) are increasingly used for detailed cognitive diagnosis.
• Existing research on DCM model fit primarily focuses on global fit, neglecting item-level diagnostics.
• There is a need for robust item-level fit checking procedures within DCM frameworks.

Purpose of the Study:

• To develop and evaluate item-level fit checking procedures for the Deterministic Input; Noisy 'And' gate (DINA) model.
• To compare the performance of classical goodness-of-fit statistics against posterior predictive model checking for item fit.
• To investigate the impact of accounting for latent attribute uncertainty on item fit detection.

Main Methods:

• Proposed two item-level fit approaches: classical goodness-of-fit statistics (chi-square, power-divergence) with Expectation-Maximization (EM) estimation, and posterior predictive model checking (PPMC) with Markov chain Monte Carlo (MCMC) estimation.
• Incorporated Stone's method to address uncertainty in latent attribute estimation for both approaches.
• Conducted a simulation study with varied factors to assess the detection rates of misfit items.

Main Results:

• Both proposed approaches demonstrated promise when Stone's method was applied.
• The classical goodness-of-fit approach exhibited a significantly higher detection rate for misfit items compared to the PPMC method.
• Stone's method improved the performance of both fit-checking strategies by accounting for latent attribute uncertainty.

Conclusions:

• Item-level fit checking is crucial for refining diagnostic classification models like DINA.
• Classical goodness-of-fit statistics, particularly when combined with Stone's method, offer a more effective strategy for detecting problematic items than PPMC.
• Future research should further explore and validate these item-level fit procedures in applied settings.