Learning to live with sampling variability: Expected replicability in partial correlation networks

Area of Science:

• Network psychometrics
• Psychological science
• Statistical modeling

Background:

• Replicability is a growing concern in network psychometrics.
• Methodological issues like p-hacking and statistical model limitations in partial correlation networks are suspected causes of unreliable findings.
• Sampling variability in partial correlations can create an illusion of unreliability and reduce statistical power.

Purpose of the Study:

• To introduce a novel methodology for deriving expected network replicability (ENR).
• To assess the inherent reliability of partial correlation networks.
• To identify factors that reduce network replicability and propose strategies for planning future replications.

Main Methods:

• Developed a new analytical method to calculate ENR using the Poisson-binomial distribution.
• Applied the ENR method to various datasets from the network literature using different partial correlation coefficients (Pearson, Spearman, Kendall, polychoric).
• Modeled replication processes to estimate expected replicability.

Main Results:

• Partial correlation networks do not appear to have inherent limitations regarding replicability; current estimates align with ENR.
• Replicability can be reduced by transitioning from continuous to ordinal data with few categories and by applying multiple comparison corrections.
• The proposed ENR method can be used to plan network replication studies.

Conclusions:

• The study provides a robust framework for assessing and planning network replicability in psychometrics.
• Recommendations include adopting gold-standard replication assessment methods and explicitly considering Type I and Type II error rates.
• The ENR computation method is available in the R package GGMnonreg.