When Do Interaction/Moderation Effects Stabilize in Linear Regression?

Area of Science:

• Statistics
• Psychometrics
• Social Sciences

Background:

• Two-way interaction effects in linear regression are crucial for understanding complex relationships.
• Estimating and testing interactions for statistical significance is often difficult due to small effect sizes and low reliability.

Purpose of the Study:

• To establish stability thresholds for two-way interactions between continuous variables using Monte Carlo simulations.
• To investigate the influence of reliability, main effect size, collinearity, and interaction effect size on interaction estimate stability.

Main Methods:

• Utilized Monte Carlo simulations to assess the stability of two-way interaction estimates.
• Defined stability using modified corridor and point of stability metrics.
• Examined various combinations of predictor reliability, main effect size, collinearity, and interaction effect size.

Main Results:

• Interaction estimate stability is primarily determined by sample size and predictor reliability.
• Realistic psychology field studies require a sample size of n = 3,800 for stability, with 72% statistical power.
• Small sample sizes (n <= 100) led to a high percentage (11-45%) of incorrectly signed interaction estimates.

Conclusions:

• Many published interaction findings in psychology may be unstable due to insufficient sample size and predictor reliability.
• Analyses with highly reliable predictors (e.g., experimental group assignment) may stabilize at lower sample sizes.
• Researchers should ensure adequate sample size and reliability before conducting two-way interaction tests, especially when hypotheses are pre-specified.