Assessing inter-rater reliability when the raters are fixed: Two concepts and two estimates.

Area of Science:

• Statistics
• Psychometrics
• Biostatistics

Background:

• Intraclass correlation (ICC) is a standard metric for assessing inter-rater reliability.
• The seminal 1979 Shrout and Fleiss paper introduced three statistical models for balanced inter-rater reliability data.
• Models 1 and 2 assumed an infinite population of raters, while Model 3 considered the sample raters as the entire population.

Purpose of the Study:

• To investigate the applicability and interpretation of ICC estimates from the first two Shrout and Fleiss models within the framework of the third model.
• To provide a clearer understanding of ICC in situations where the rater sample represents the complete population of interest.

Main Methods:

• The study re-examines the two distinct ICC estimates derived from the first two Shrout and Fleiss models.
• These estimates are then applied to data structured according to the third Shrout and Fleiss model.
• The different interpretations of these estimates in the context of the third model are discussed.

Main Results:

• The two ICC estimates developed for infinite rater populations are shown to be applicable to the third model, where raters are considered a finite population.
• The study elucidates the distinct interpretations that arise when applying these estimates within the finite rater population context.

Conclusions:

• The findings extend the utility of established ICC estimation methods to scenarios involving finite rater populations.
• This work offers a more nuanced understanding of inter-rater reliability assessment using ICC, particularly when generalizing beyond the sampled raters is not intended.