Related Experiment Videos
[How to design an interobserver reliability study for categorical variables: a literature review]
Ana Beatriz Rusconi Lagarrigue1, Sebastián Andrés Sguiglia Schütz2, Victoria Guarnieri1
1Hospital Italiano de Buenos Aires.
Introduction:
On many occasions, it is necessary to establish the degree of agreement between two or more observers, either as an end in itself or as an intermediate step in a research project. In this type of study, choosing the appropriate design is crucial, as it influences the applicability and generalizability of the methods.
Objectives:
In this article, we will present—using a real case from a study still in progress—the methodology for estimating the degree of agreement between two or more observers regarding a categorical variable.
Materials And Methods:
To define the coefficients used, a bibliographic search was conducted in PubMed and Google Scholar. Results: We propose the use of weighted Fleiss's kappa and Gwet's AC2 as agreement metrics for this problem. AC2 is a new coefficient that overcomes some of the limitations associated with kappa, particularly the paradoxes described by Feinstein and Cichetti.
Conclusion:
In this article, we present the design of a study to evaluate the degree of interobserver agreement and describe the methodology in detail, including the characteristics of the judges, the subjects, and the categories to be evaluated, the choice of the coefficient of agreement and its interpretation, and the calculation of confidence intervals.