Updated guidelines on selecting an intraclass correlation coefficient for interrater reliability, with applications...

Area of Science:

• Psychometrics
• Observational Research Methods
• Statistical Reliability

Background:

• Intraclass correlation coefficients (ICCs) are used for interrater reliability (IRR) in observational studies.
• Existing guidelines for selecting ICCs have limitations, including handling incomplete designs and clarifying error variance.
• Misunderstandings regarding fixed versus random raters complicate ICC selection.

Purpose of the Study:

• To provide updated, generalizability theory-based guidelines for selecting ICCs for IRR.
• To address limitations of existing guidelines, including incomplete and unbalanced designs.
• To clarify the interpretation of ICCs and the role of raters (fixed vs. random).

Main Methods:

• Utilized generalizability theory (GT) to develop updated ICC selection guidelines.
• Developed a flowchart to guide researchers through four key choices in ICC selection.
• Applied the flowchart to three empirical examples from clinical and developmental research.

Main Results:

• The updated guidelines are applicable to both complete and incomplete observational designs.
• The study challenges the notion of raters being fixed, suggesting they should seldom be considered fixed.
• Clarified ICC interpretation for unbalanced and incomplete designs.

Conclusions:

• Researchers should carefully consider ICC selection based on design and error variance.
• Raters are more appropriately conceptualized as random effects in most IRR contexts.
• Provides practical guidance for reporting, interpreting, and estimating ICCs for IRR.