Assessing agreement with multiple raters on correlated kappa statistics

Area of Science:

• Biostatistics
• Clinical Research Methodology
• Medical Diagnostics

Background:

• Assessing diagnostic agreement among clinicians is crucial in clinical studies.
• Existing methods often analyze agreement between two clinicians or among multiple clinicians under a single condition.
• Limited research addresses agreement among multiple clinicians evaluating the same patients under two distinct conditions.

Purpose of the Study:

• To develop and evaluate a statistical method for assessing diagnostic agreement among multiple clinicians under two different conditions.
• To address the gap in analyzing correlated kappa statistics in complex clinical study designs.
• To provide a robust tool for evaluating inter-rater reliability in multi-condition diagnostic scenarios.

Main Methods:

• Utilized the intraclass kappa statistic for nominal scale agreement.
• Derived an explicit variance formula for the difference between correlated kappa statistics.
• Conducted hypothesis testing for the equality of kappa statistics.
• Employed simulation studies to assess method performance.

Main Results:

• The proposed method demonstrates good performance with realistic sample sizes.
• The new approach may be superior to methods that ignore the measurement dependence structure.
• The statistical approach effectively handles correlated kappa statistics in multi-clinician, multi-condition designs.

Conclusions:

• The developed statistical method provides a reliable way to assess diagnostic agreement in complex clinical settings.
• This approach enhances the understanding of inter-rater reliability when multiple clinicians assess patients under varying conditions.
• The method's utility is validated through simulations and demonstrated in an eosinophilic esophagitis (EoE) study.