Assessing method agreement for paired repeated binary measurements administered by multiple raters.

Area of Science:

• Medical Statistics
• Biometry
• Clinical Trial Methodology

Background:

• Method comparison studies are crucial for validating new medical and clinical assessments.
• Rater variability significantly impacts measurement consistency in clinical assessments, especially in large-scale studies.
• Existing agreement metrics may not adequately address simultaneous assessment of method and rater agreement for repeated binary measurements.

Purpose of the Study:

• To propose a novel model-based approach for assessing agreement between two measurement methods for paired repeated binary data.
• To simultaneously evaluate agreement between measurement methods and agreement among raters.
• To provide a framework for determining the interchangeability of new measurement methods in clinical settings.

Main Methods:

• Development of a model-based approach using generalized linear mixed models (GLMMs).
• Testing the equality of fixed effects of methods to decide on interchangeability.
• Extension of established agreement assessment approaches (e.g., Bland-Altman, Cohen's kappa) for repeated binary measurements using latent variables within GLMMs.

Main Results:

• The proposed GLMM-based approach effectively assesses method agreement for paired repeated binary measurements.
• Simulation studies confirmed the model's ability to handle simultaneous assessment of method and rater agreement.
• Real clinical data analysis demonstrated the practical utility of the approach in evaluating delirium screening methods.

Conclusions:

• The novel model-based approach provides a robust method for assessing agreement in paired repeated binary measurements, accounting for rater variability.
• This approach enhances the reliability of method comparison studies in medical and clinical fields.
• The developed methods support informed decisions regarding the interchangeable use of new clinical assessment tools.