Sample size determination for hypothesis testing on the intraclass correlation coefficient in a two-way analysis of...

Area of Science:

• Psychometrics
• Medical Statistics
• Psychology

Background:

• Reliability evaluation is crucial for accurate diagnosis and treatment in psychology and medicine.
• The intraclass correlation coefficient for agreement (ICCa) is a key metric when using the same raters.
• Existing sample size methods primarily focus on confidence interval width, with limited options for hypothesis testing.

Purpose of the Study:

• To address the gap in sample size determination for hypothesis testing of ICCa.
• To propose procedures ensuring adequate statistical power to test if ICCa surpasses a predefined threshold.
• To evaluate and recommend sample size strategies based on robust ICCa confidence interval methods.

Main Methods:

• Comparison of various confidence interval methods for ICCa.
• Development of sample size procedures utilizing the lower confidence limit of the best-performing ICCa methods.
• Empirical evaluation of proposed procedures' power under diverse parameter settings.

Main Results:

• Identification of optimal confidence interval methods for ICCa.
• Validation of proposed sample size procedures through power analysis.
• Demonstration of improved statistical power for ICCa hypothesis tests with the new methods.

Conclusions:

• The proposed sample size procedures effectively enhance statistical power for ICCa hypothesis testing.
• Researchers can now determine adequate sample sizes for reliability studies with hypothesis testing goals.
• An interactive R shiny app is available for practical implementation of these sample size determination procedures.