Many nonnormalities, one simulation: Do different data generation algorithms affect study results?

Area of Science:

• Statistics
• Psychometrics
• Meta-science

Background:

• Monte Carlo simulation studies are crucial for statistical methodology but often lack replication.
• Replicability failures in simulations can stem from various factors, impacting the reliability of statistical guidance.
• Few simulation studies are ever replicated, limiting the validation of their findings.

Purpose of the Study:

• To replicate a highly cited 1996 simulation study on the robustness of normal theory maximum likelihood (ML)-based chi-square fit statistics under multivariate nonnormality.
• To examine the generalizability of the original study's findings across different nonnormal data generation algorithms.
• To assess the impact of varying data generation methods on the robustness conclusions of ML-based fit statistics.

Main Methods:

• Conducted a meta-scientific study by replicating a specific simulation study (Curran et al., 1996).
• Employed multiple nonnormal data generation algorithms to test the generalizability of original findings.
• Compared replication results with original findings and analyzed discrepancies across algorithms.

Main Results:

• Replication results were generally consistent with the original study, but several differences were noted.
• Generalizability results were mixed; only two findings held across all examined algorithms.
• The independent generator (IG) algorithm yielded substantially different results, suggesting ML robustness to nonnormality for the tested factor model.

Conclusions:

• Extant methodological recommendations derived from simulations may not be universally applicable when multiple data generation algorithms exist.
• Researchers should consider multiple data generation approaches to enhance the generalizability of simulation study findings.
• The choice of data generation algorithm can significantly influence the robustness conclusions of statistical methods like ML.