Coefficients of agreement for fixed observers

Area of Science:

• Biostatistics
• Medical Statistics
• Psychometrics

Background:

• Intraclass correlation coefficients (ICCs) are standard for evaluating agreement between observers or methods on continuous scales.
• Current ICC methods often rely on the two-way analysis of variance (ANOVA) model, which imposes restrictive assumptions.

Purpose of the Study:

• To address limitations of the ANOVA model in assessing observer agreement.
• To introduce and discuss the concept of observer relational agreement for deriving ICCs without ANOVA's strict assumptions.
• To propose a new agreement coefficient that avoids the unrealistic assumption of 'chance agreement'.

Main Methods:

• Critique of ANOVA model assumptions (e.g., homogeneity of error variances and observer correlations).
• Application of observer relational agreement from social sciences to derive ICCs.
• Comparison of concordance correlation coefficient (CCC) with observer agreement definitions.
• Development of a novel agreement coefficient not based on chance agreement.

Main Results:

• The ANOVA model's assumptions are often inadequate for real-world observer agreement scenarios.
• Observer relational agreement provides a framework for calculating ICCs without restrictive ANOVA assumptions.
• The CCC, while popular, relies on an unrealistic assumption of observer independence ('chance agreement').
• A new coefficient is presented that bypasses the 'chance agreement' concept.

Conclusions:

• Traditional ICCs based on ANOVA may yield misleading results due to unmet assumptions.
• Observer relational agreement offers a more flexible and robust approach to evaluating measurement consistency.
• The proposed new coefficient provides a more realistic alternative for assessing observer agreement in biomedical research.