Parameter Estimation Accuracy of the Effort-Moderated Item Response Theory Model Under Multiple Assumption Violations

Area of Science:

• Educational Measurement
• Psychometrics
• Statistical Modeling

Background:

• Low-stakes testing environments increasingly face validity threats due to low test-taking effort.
• The effort-moderated item response theory (EM-IRT) model is a proposed solution to address noneffortful responses by treating them as missing data.
• Previous research has not fully evaluated EM-IRT's performance under violations of its core assumptions.

Purpose of the Study:

• To investigate the impact of violating EM-IRT assumptions on item and mean ability parameter recovery.
• To compare the performance of EM-IRT against the traditional two-parameter logistic (2PL) model under simulated assumption violations.

Main Methods:

• A simulation study was conducted to examine parameter recovery under violations of two key EM-IRT assumptions: random occurrence of noneffortful responding and its independence from examinee ability.
• Item and mean ability parameters were estimated using both EM-IRT and 2PL models across various simulated conditions.
• An accompanying empirical study was used to contextualize simulation findings.

Main Results:

• The EM-IRT model demonstrated robust item parameter estimates when the assumption of random noneffortful responding was violated.
• Violations to the assumption of noneffortful responding being unrelated to ability introduced bias in EM-IRT item parameter estimates, though less than in the 2PL model.
• Both EM-IRT and 2PL models showed similar performance for mean ability estimates, with notable bias observed when the second assumption was violated, particularly under extreme conditions.

Conclusions:

• The EM-IRT model offers superior item parameter estimation compared to the 2PL model, even when its assumptions are violated under realistic conditions.
• EM-IRT provides comparable mean ability parameter estimates to the 2PL model, with potential bias under specific, possibly extreme, violation scenarios.
• The findings support the utility of EM-IRT in addressing validity threats posed by low effort in low-stakes testing contexts.