Recalibrating expectations about effect size: A multi-method survey of effect sizes in the ABCD study

Area of Science:

• Psychology
• Developmental Psychology
• Psychometrics

Background:

• Established heuristics for interpreting effect sizes (e.g., Cohen's guidelines) may not accurately reflect typical findings in psychological research.
• There is a need to re-evaluate these benchmarks using empirical data from large, diverse samples.

Purpose of the Study:

• To describe the distribution of effect sizes across multiple instruments in a large sample of children.
• To identify factors influencing effect size distributions.
• To establish empirical benchmarks for effect sizes within the Adolescent Brain and Cognitive Development Study (ABCD) dataset.

Main Methods:

• Pearson's correlations were computed for 161 variables from the ABCD Study baseline data.
• Effect size distributions were analyzed across different variable types, statistical thresholds, and covariate strategies.
• Analyses were conducted to mimic real-world data analysis conditions.

Main Results:

• The median in-sample effect size was .03 (Q1=.01, Q3=.07) in initial analyses.
• Effect sizes were smaller when considering associations across different instruments, content domains, reporters, and when controlling for sociodemographic factors.
• Effect sizes were larger when thresholding for statistical significance, with a median of .05 (Q1=.03, Q3=.09) in analyses mimicking real-world conditions.

Conclusions:

• Empirically derived effect sizes from the ABCD dataset are smaller than those suggested by traditional heuristics.
• The findings highlight the need for updated, empirically grounded benchmarks for interpreting effect sizes in psychological research.
• This study provides specific examples from the ABCD dataset to serve as new benchmarks.