How to Estimate Intraclass Correlation Coefficients for Interrater Reliability from Planned Incomplete Data.

Area of Science:

• Behavioral Science
• Psychometrics
• Statistical Modeling

Background:

• Interrater reliability (IRR) is crucial for observational data, often assessed using Intraclass Correlation Coefficients (ICCs).
• Traditional ICC estimation methods using ANOVA are problematic with incomplete data, common in planned missing observational designs.
• Behavioral research frequently employs planned missing designs, necessitating robust ICC estimation techniques for incomplete datasets.

Purpose of the Study:

• To compare the computational accuracy and feasibility of three novel ICC estimation methods for planned incomplete observational data.
• To identify the most reliable method for estimating ICCs in the presence of missing data within behavioral research contexts.

Main Methods:

• Simulated planned incomplete data to mimic real-world observational studies.
• Evaluated three estimation methods: Bayesian hierarchical linear models (MCMC), maximum likelihood (ML) for random-effects models, and ML for common-factor models.
• Assessed computational accuracy (bias, RMSE, coverage) and feasibility (convergence, time).

Main Results:

• Maximum likelihood estimation of random-effects models demonstrated superior performance across all evaluated criteria.
• This method showed better accuracy in point and variability estimates and higher coverage rates compared to alternatives.
• The study provides R code for practical application of these advanced ICC estimation techniques.

Conclusions:

• Maximum likelihood estimation of random-effects models, particularly with Monte Carlo confidence intervals, is the preferred method for ICC estimation with incomplete observational data.
• The findings offer practical guidance for researchers in behavioral sciences dealing with planned missing data.
• Availability of R code facilitates the implementation of these improved statistical methods in future research.