Modelling patterns of agreement and disagreement

Area of Science:

• Statistics
• Psychometrics
• Data Analysis

Background:

• Observer agreement is crucial for reliable data collection.
• Quantifying inter-observer agreement for categorical data presents statistical challenges.
• Existing methods may not fully capture the nuances of rater agreement.

Purpose of the Study:

• To provide a comprehensive overview of statistical modeling techniques for observer agreement.
• To emphasize methods applicable to both nominal and ordinal categorical response scales.
• To discuss models that address both the strength and pattern of agreement.

Main Methods:

• Review of statistical models including cell-probability models (e.g., Cohen's kappa), loglinear models (quasi-independence, quasi-symmetry), latent class models, and Rasch models.
• Focus on modeling inter-observer agreement for categorical responses.
• Discussion of how models address association strength and marginal distribution similarity.

Main Results:

• Various statistical models offer different approaches to quantifying observer agreement.
• Models can disentangle the strength of association from the similarity of marginal distributions.
• Latent class and Rasch models provide more nuanced insights into rater behavior and agreement patterns.

Conclusions:

• A range of statistical models are available for analyzing observer agreement in categorical data.
• The choice of model depends on the specific research question and data characteristics.
• Understanding these models enhances the reliability and validity of studies involving multiple observers.