Model modifications in covariance structure analysis: the problem of capitalization on chance

Area of Science:

• Statistics
• Psychometrics
• Quantitative Psychology

Background:

• Covariance structure modeling is frequently used in statistical analysis.
• When an initial model does not fit data well, model modification is common.
• Data-driven modifications raise concerns about generalizability to new samples or populations.

Purpose of the Study:

• To investigate the generalizability of covariance structure models modified based on sample data.
• To assess the reliability and consistency of data-driven model modifications.
• To compare data-driven modification strategies with the use of alternative a priori models.

Main Methods:

• The study discusses the issue of data-driven model modification in covariance structure modeling.
• Empirical exploration was conducted using sampling studies with two large datasets.
• Cross-validation techniques were employed to evaluate model generalizability.

Main Results:

• Model modifications based on sample data were found to be inconsistent across repeated samples.
• Cross-validation results exhibited erratic behavior, indicating poor generalizability.
• Capitalizing on chance characteristics of the data significantly impacts model stability.

Conclusions:

• Skepticism regarding the generalizability of models resulting from data-driven modifications is warranted.
• Data-driven modifications can lead to overfitting and unreliable findings.
• The use of alternative a priori models is recommended as a more robust strategy.