Data with hierarchical structure: impact of intraclass correlation and sample size on type-I error

Area of Science:

• Statistics
• Experimental Design
• Psychometrics

Background:

• Hierarchical data structures are common in various scientific fields.
• Standard statistical methods like least squares (LS) are often applied to analyze such data.
• LS analyses assume data independence, which is violated in hierarchical structures, potentially leading to biased results.

Purpose of the Study:

• To investigate the impact of using least squares methods on Type-I error rates in hierarchical data analysis.
• To evaluate the severity of Type-I error inflation in the context of "groups nested under treatment" experimental designs.
• To assess the effectiveness and limitations of the Kish (1965) correction for hierarchical data.

Main Methods:

• Simulations were conducted to examine Type-I error rates under varying conditions of intraclass correlation and sample size.
• Least squares analyses (e.g., ANOVAs, linear regressions) were applied to simulated hierarchical datasets.
• The Type-I error rates were compared to nominal rates and evaluated with and without the Kish (1965) correction.

Main Results:

• Least squares analyses of hierarchical data result in severely inflated Type-I error rates, departing significantly from nominal levels.
• The degree of Type-I error inflation is influenced by intraclass correlation and sample size.
• The Kish (1965) correction showed limitations in adequately controlling Type-I error rates in the simulated scenarios.

Conclusions:

• The application of least squares methods to hierarchical data, particularly in group-based experimental designs, poses a significant risk of inflated Type-I errors.
• Researchers must be cautious when interpreting results from LS analyses of hierarchical data.
• Recommendations are provided for appropriate data collection and analysis strategies for hierarchical data structures.