Do method effects generalize across traits (and what if they don't)?

Area of Science:

• Psychometrics
• Quantitative Psychology
• Statistical Modeling

Background:

• Multitrait-multimethod (MTMM) data are frequently analyzed using single-indicator confirmatory factor analysis (CFA-MTMM) models.
• These models often rely on an untested, restrictive assumption that method biases correlate perfectly across traits for a given method.

Purpose of the Study:

• To evaluate the validity of the assumption of perfectly general method effects in single-indicator CFA-MTMM models.
• To investigate the consequences of violating this assumption using simulated data and a review of published studies.

Main Methods:

• Reviewed 20 published applications of multiple-indicator CFA-MTMM models, which allow for testing the assumption of method effect correlations.
• Utilized simulated data and the CT-C(M-1) approach to demonstrate the impact of violating the assumption of perfectly general method effects.

Main Results:

• Analysis of 111 heterotrait-monomethod correlations revealed substantial variation (mean = .52), with most values significantly less than 1.
• Violations of the perfectly general method effects assumption are common, hard to detect via model fit, and introduce bias in validity, reliability, and method factor estimates.

Conclusions:

• The assumption of perfectly general method effects in single-indicator CFA-MTMM models is frequently violated and problematic.
• Researchers are strongly advised to transition to multiple-indicator CFA-MTMM models to ensure more accurate and valid results in MTMM analyses.