Estimating Item Wording Effects in Self-Report Measures with Generalizability Theory-Based SEMs: Illustrations Using...

Area of Science:

• Psychological Measurement
• Structural Equation Modeling
• Psychometrics

Background:

• Item wording effects pose challenges in Likert-style, self-report questionnaires for psychological traits.
• Current methods often correlate item uniquenesses or use method factors, but rarely quantify wording effect magnitudes distinctly from measurement error.

Purpose of the Study:

• To demonstrate the utility of generalizability theory-based structural equation models for disentangling construct, item wording, and measurement error effects.
• To provide a robust framework for analyzing measurement variance in psychological assessments.

Main Methods:

• Employed generalizability theory-based structural equation model designs.
• Analyzed data from a large sample (n=1,796) of college students using the Self-Description Questionnaire-III on two occasions.
• Provided R code for analyzing generalizability theory and conventional congeneric structural equation models.

Main Results:

• Results highlighted the necessity of separating construct, item wording, specific-factor, transient, and random-response error effects for each subscale.
• Generalizability theory techniques proved effective in distinguishing these variance components.
• In comprehensive models, targeted constructs explained the most variance, followed by random-response, transient, specific-factor error, and item wording effects.

Conclusions:

• Generalizability theory-based structural equation modeling offers a superior method for analyzing measurement properties in psychological questionnaires.
• Accurate assessment requires distinguishing between construct variance, item wording effects, and multiple sources of measurement error.