Identifying measurement disturbance effects using Rasch item fit statistics and the Logit Residual Index

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Rasch item fit statistics are crucial for evaluating item performance in psychometric models.
• Guessing behavior in test-takers can potentially influence item fit evaluations.
• The Logit Residual Index (LRI) is a proposed statistic for detecting item misfit.

Purpose of the Study:

• To investigate the impact of guessing on Rasch item fit statistics (weighted total, unweighted total, unweighted between).
• To evaluate the sensitivity of the Logit Residual Index (LRI) to item misfit under varying guessing conditions.
• To compare the performance of different fit statistics and the LRI in simulated dichotomous data.

Main Methods:

• A Monte Carlo simulation study was employed.
• Dichotomous data were simulated with 100 items and 100 persons.
• Three guessing levels (0%, 25%, 50%) and two item difficulty distributions (normal, uniform) were used.

Main Results:

• No significant differences in mean Rasch item fit statistics were observed across guessing levels for either distribution.
• Mean item scores differed significantly for uniform, but not normal, item difficulties as guessing increased.
• The LRI showed higher sensitivity to large positive unweighted total fit statistic values, indicating item misfit.

Conclusions:

• Rasch item fit statistics appear robust to guessing in this simulated dichotomous data context.
• The LRI is a valuable tool for detecting item misfit, particularly linear trends in residuals, suggesting differential item functioning.
• The unweighted total fit statistic, when large, significantly impacts LRI values, highlighting its utility in identifying problematic items.