Good Fit Is Weak Evidence of Replication: Increasing Rigor Through Prior Predictive Similarity Checking.

Area of Science:

• Psychological research methodology
• Quantitative psychology
• Computational modeling

Background:

• Replication is crucial for validating psychological theories, including model-based research.
• Goodness of fit (GOF) is commonly used to assess replication success but may obscure critical discrepancies.
• Existing methods may not adequately capture the similarity between original and replication study data patterns and parameter estimates.

Purpose of the Study:

• To introduce Bayesian prior predictive similarity checking as a rigorous method for evaluating model-based replication.
• To demonstrate the limitations of Goodness of Fit (GOF) in assessing empirical replication.
• To compare GOF with Bayesian prior predictive similarity checking using real-world data.

Main Methods:

• Application of Bayesian prior predictive similarity checking to original and replication data from the National Comorbidity Survey.
• Evaluation of data patterns and parameter estimates for similarity between studies.
• Comparison of results obtained from GOF and the proposed similarity checking method.

Main Results:

• Both original and replication datasets showed excellent GOF.
• Bayesian prior predictive similarity checks frequently indicated a lack of close or approximate empirical replication.
• Discrepancies were particularly evident in covariance patterns and indicator thresholds.

Conclusions:

• GOF can be misleading as a sole indicator of successful replication.
• Bayesian prior predictive similarity checking provides a more sensitive and rigorous evaluation of empirical replication.
• Recommendations include adopting registered reports and utilizing similarity checking for model-based research.