Unbalanced 2 x 2 factorial designs and the interaction effect: a troublesome combination

Area of Science:

• Statistics
• Data Analysis
• Research Methodology

Background:

• Analysis of Variance (ANOVA) is a statistical method used to analyze differences among group means.
• Unbalanced datasets, where group sizes differ, can pose challenges in ANOVA.
• Sums of Squares Type II (SS II) and Type III (SS III) are two common methods for correcting ANOVA for unbalanced data.

Purpose of the Study:

• To compare the statistical power of SS II and SS III ANOVA methods in unbalanced 2x2 datasets.
• To evaluate the performance of these methods when interaction effects are present.
• To determine if these methods provide satisfactory power and accurate re-estimation of effects.

Main Methods:

• A power study was conducted using unbalanced and balanced 2x2 datasets (N=120).
• Datasets were constructed assuming the alternative hypothesis (H1) was true, with effects originating from treatment groups or contrasting contributions from treatment and control groups.
• Analyses involved comparing rejection rates of the null hypothesis (H0) for main effects and interactions using SS II and SS III.

Main Results:

• SS II generally showed higher power but yielded highly variable results across different dataset constructions.
• SS III consistently demonstrated slightly lower power than SS II but provided more stable and predictable outcomes.
• When effects were equally contributed by treatment and control groups, interaction effects were satisfactorily re-estimated. SS III showed lower rejection rates for H0 of main effects in the presence of interaction compared to balanced datasets.
• In scenarios with effects solely in treatment groups, SS II was applicable for moderate to strong interaction effects, while SS III performed slightly better with true interactions.

Conclusions:

• Neither SS II nor SS III consistently allowed for satisfactory re-estimation of unique interaction effects in all tested scenarios.
• SS II was only superior when interaction effects could be excluded, whereas SS III performed marginally worse in such cases.
• Overall, SS III is recommended for general application due to its consistent performance, despite slightly lower power (1-5% lower rejection rates than balanced datasets), while SS II's results varied too widely for broad use.