A unified approach for assessing agreement for continuous and categorical data

Area of Science:

• Biostatistics
• Statistical Modeling
• Data Analysis

Background:

• Accurate measurement agreement is crucial in various scientific fields.
• Existing methods for assessing inter-rater and intra-rater reliability have limitations, especially with multiple readings.
• There is a need for comprehensive indices that capture agreement, precision, and accuracy across different data types.

Purpose of the Study:

• To propose novel Concordance Correlation Coefficient (CCC) indices for measuring agreement among k raters with m readings per subject.
• To develop indices for continuous and categorical data, including coverage probability (CP) and total deviation index (TDI) for normal data.
• To assess intra-rater, inter-rater, and total agreement, precision, and accuracy within a unified statistical framework.

Main Methods:

• Development of generalized Concordance Correlation Coefficient (CCC) indices for multiple raters and readings.
• Application of a two-way mixed model to express indices as functions of variance components.
• Utilization of the Generalized Estimating Equations (GEE) method for estimation and inference.

Main Results:

• The proposed CCC indices provide a unified approach, reducing to previously established methods under specific conditions (e.g., m→∞, m=1, k=2).
• The framework allows for the assessment of intra-rater, inter-rater, and total agreement, precision, and accuracy.
• Indices for continuous and categorical data are derived, including CP and TDI for normal distributions.

Conclusions:

• The proposed CCC indices offer a flexible and comprehensive tool for evaluating measurement agreement and precision.
• This unified approach simplifies the assessment of reliability across diverse experimental designs and data types.
• The methods provide a robust statistical foundation for agreement studies in biostatistics and related fields.