Comparison of confidence interval methods for an intra-class correlation coefficient (ICC)

Area of Science:

• Biostatistics
• Reliability Analysis
• Measurement Science

Background:

• The intraclass correlation coefficient (ICC) is a vital metric in biomedical research for assessing measurement reproducibility across raters, labs, technicians, or devices.
• Accurate ICC estimation is essential for inter-rater reliability studies, where high ICC values indicate minimal noise relative to patient variability.
• Confidence or Bayesian credible intervals for ICC are standard summaries, with construction relying on frequentist or Bayesian approaches.

Purpose of the Study:

• To evaluate three methods for constructing ICC intervals within a two-way, crossed, random effects model without interaction: Generalized Confidence Interval (GCI), Modified Large Sample (MLS), and Bayesian Noninformative Prior (NIB).
• To provide guidance on selecting interval construction methods based on study design, sample size, and data normality.
• To compare the coverage probabilities and widths of these interval methods.

Main Methods:

• The study analyzed three interval construction methods: GCI, MLS, and NIB, applied to a two-way, crossed, random effects model without interaction.
• Performance was assessed by comparing coverage probabilities and interval widths.
• Guidance for method selection was developed considering study design, sample size, and data normality.

Main Results:

• Interval method selection requires careful consideration, as estimates may not always behave as expected in the two-way, crossed, random effects model without interaction.
• Methods generally perform well with numerous factor levels, but significant differences emerge when one or more factors have limited levels.
• All methods demonstrated a lack of robustness to subtle normality violations, particularly with limited sample sizes.

Conclusions:

• Decision rules and R-language software are provided for practical ICC interval construction in the specified model.
• All interval methods perform comparably with normal data and sufficient factor levels.
• MLS and GCI methods are superior to NIB for normally distributed or near-normal data when a factor has limited levels; however, no method performs well with limited factor levels and markedly non-normal data.