Measuring Agreement in Diagnostics: A Practical Guide for Researchers

Area of Science:

• Biostatistics
• Clinical Epidemiology
• Medical Informatics

Background:

• Clinical assessments require accurate interpretation for patient care.
• Existing reliability and agreement studies in intrapartum fetal heart rate monitoring have methodological limitations.
• Confusion exists between agreement and reliability, calculation methods for multiple raters, and reporting of confidence intervals.

Purpose of the Study:

• To clarify computation and interpretation of agreement measures for binary outcomes.
• To provide guidance on statistical inference and sample size calculations for agreement studies.
• To enhance the methodological quality of diagnostic test agreement studies.

Main Methods:

• Demonstration using a motivating example of five obstetricians assessing 20 CTGs (cardiotocography).
• Explanation of agreement definitions, computation, and interpretation in various scenarios.
• Discussion of the relationship between agreement, reliability, intra-observer, and inter-observer studies.

Main Results:

• Emphasis on proportion of agreement, proportion of specific agreement, and kappa coefficients.
• A developed shiny application to assist researchers in agreement studies.
• The work complements existing reporting guidelines like GRRAS, QAREL, and STARD.

Conclusions:

• Improved understanding and application of agreement measures in clinical research.
• Enhanced methodological rigor in studies evaluating diagnostic test agreement.
• Facilitation of more reliable diagnostic assessments and better patient outcomes.