Disentangling substantive and method variance in mixed-worded scales: An empirical application of the random...

Addressing method variance in mixed-worded scales poses a persistent challenge in psychometric research. This study evaluates the effectiveness of the random intercept item factor analysis (RIIFA) in distinguishing substantive variance from method variance, specifically the wording effect, using the Short Grit Scale (Grit-S) as an applied example. Two independent UK samples were analyzed using exploratory graph analysis (EGA), parallel analysis (PA), and confirmatory factor analysis (CFA) with and without a random intercept factor. Traditional dimensionality assessment methods and RIIFA-based counterparts were compared to evaluate their ability to control for spurious factor emergence due to the wording effect. Consistent with our hypotheses, Study 1 (N = 977) confirmed that traditional retention methods overestimated the number of factors, whereas RIIFA techniques provided unidimensional and more stable solutions, as supported by bootstrap analyses. Study 2 (N = 496) showed that the CFA model incorporating a random intercept factor achieved the best balance between parsimony and fit (root mean square error of approximation [RMSEA] = .048 [.022-.072]; comparative fit index [CFI] = .984; Tucker-Lewis index [TLI] = .974; standardized root mean square residual [SRMR] = .027), while yielding a hierarchical omega of .84. These findings indicate that RIIFA reallocates the explained variance, mitigating artificial bidimensionality and enhancing the structural validity of the scale. RIIFA offers a robust psychometric solution for handling method variance in mixed-worded scales, improving latent structure interpretability. In sum, we recommend its application in cases where wording effects threaten the validity of psychometric measurements.