Item Response Theory Models for Wording Effects in Mixed-Format Scales

Area of Science:

• Psychometrics
• Educational Measurement
• Item Response Theory

Background:

• Mixed-format scales often include both positively and negatively worded items.
• Standard reverse recoding may not adequately address wording effects.
• Existing methods for handling wording effects have limitations.

Purpose of the Study:

• To evaluate the effectiveness of reverse coding for negatively worded items.
• To assess the magnitude of wording effects in mixed-format scales.
• To introduce and demonstrate a bi-factor item response theory (IRT) approach for analyzing wording effects.

Main Methods:

• Utilized bi-factor item response theory (IRT) models.
• Tested the assumption of reverse coding in mixed-format scales.
• Applied the models to empirical data from large-scale educational assessments (PISA, TIMSS).

Main Results:

• Wording effects were found to be substantial in the analyzed datasets.
• Ignoring wording effects led to overestimated test reliability.
• Failure to account for wording effects resulted in biased person measures.

Conclusions:

• Reverse coding is insufficient to equate negatively and positively worded items.
• Bi-factor IRT models offer a robust method for quantifying wording effects.
• Addressing wording effects is crucial for accurate measurement in educational assessments.