A comparison of two indices for the intraclass correlation coefficient

Area of Science:

• Psychometrics
• Multilevel Modeling
• Statistical Analysis

Background:

• Intraclass correlation coefficients (ICC) are crucial for assessing reliability in multilevel research.
• Existing methods like ICC(1) and corrected eta-squared have different estimation approaches and performance characteristics.
• Understanding their biases and errors is vital for accurate reliability analysis.

Purpose of the Study:

• To compare the performance of ICC(1) and corrected eta-squared in estimating intraclass correlation in one-way random effects models.
• To analytically and empirically investigate the bias and mean squared error of these two indices.
• To provide guidance for researchers in selecting appropriate reliability analysis tools.

Main Methods:

• Analytical derivation showing corrected eta-squared equivalence to maximum likelihood and pairwise estimators under equal group sizes.
• Monte Carlo simulation study across various group structures to evaluate performance.
• Comparison of bias and mean squared error for ICC(1) and corrected eta-squared.

Main Results:

• Corrected eta-squared estimator is equivalent to maximum likelihood and pairwise estimators with equal group sizes.
• A mutual dominance relationship exists between truncated versions for negative values.
• Corrected eta-squared outperforms ICC(1) for small population intraclass correlation coefficients.
• ICC(1) shows a clear advantage over corrected eta-squared for medium and large population intraclass correlation coefficients.

Conclusions:

• The choice between ICC(1) and corrected eta-squared depends on the expected magnitude of the population intraclass correlation coefficient.
• Corrected eta-squared is preferred for small ICC values, while ICC(1) is better for larger values.
• This research offers practical guidelines for selecting the most accurate reliability index in multilevel research.