Modeling Wording Effects Does Not Help in Recovering Uncontaminated Person Scores: A Systematic Evaluation With...

Area of Science:

• Psychometrics
• Statistical Modeling
• Educational Measurement

Background:

• The item wording effect involves inconsistent responses to similarly-themed, oppositely-worded survey items.
• Random intercept item factor analysis (RIIFA) models have shown promise in accounting for this effect.
• The efficacy of RIIFA in recovering accurate person scores remains under-explored.

Purpose of the Study:

• To evaluate the performance of the RIIFA model in estimating uncontaminated person scores across different item wording effects.
• To compare RIIFA with traditional substantive unidimensional models under various simulation conditions.

Main Methods:

• Monte Carlo simulations were used to manipulate four variables: wording effect type (carelessness, verification difficulty, acquiescence), effect amount, sample size, and test length.
• The study analyzed model fit and the accuracy of person score recovery.
• Simulation results were validated using an empirical dataset on undergraduate academic performance.

Main Results:

• RIIFA models consistently demonstrated excellent fit, effectively accounting for item wording effects regardless of their magnitude.
• Models without the RIIFA factor showed a poorer fit as wording effects increased.
• Surprisingly, RIIFA did not outperform substantive unidimensional models in estimating uncontaminated person scores.

Conclusions:

• While RIIFA effectively addresses data variance caused by item wording effects, it does not enhance the precision of person score estimation.
• The study highlights a discrepancy between model fit and person score recovery in the presence of wording effects.
• Understanding the properties of factor models is crucial for interpreting these findings in psychometric and educational research.