Evaluation of Item Fit With Output From the EM Algorithm: RMSD Index Based on Posterior Expectations.

Area of Science:

• Psychometrics
• Statistical Modeling
• Educational Measurement

Background:

• Item response theory (IRT) relies on item fit analysis for model validity.
• Posterior expectations (pseudocounts) offer advantages for item fit, especially with missing data.
• The interpretability of the root mean squared deviation (RMSD) index needs improvement.

Purpose of the Study:

• To enhance the interpretability of the RMSD index derived from posterior expectations in IRT.
• To evaluate and compare different methods for determining optimal cutoff thresholds for item fit.
• To develop a generalizable prediction model for RMSD reference values based on sample size and test length.

Main Methods:

• Utilized poor person's posterior predictive model checking (PP-PPMC) to assess significance levels.
• Employed receiver operating characteristic (ROC) curve analysis to empirically determine optimal cutoff thresholds.
• Applied response surface analysis to create a prediction model for reference values.

Main Results:

• The cutoff threshold approach demonstrated superior performance over PP-PPMC, balancing false and true positive rates effectively.
• Optimal reference values were identified for various sample sizes and test lengths.
• The prediction model accurately generalized how reference values vary with dataset characteristics.

Conclusions:

• The study validates PP-PPMC for item fit diagnostics and introduces a practical frequentist method for deriving reference values.
• The developed prediction model allows researchers to compute dataset-specific RMSD reference values.
• This work provides a more refined approach to item fit analysis in IRT modeling.