Indexing systematic rater agreement with a latent-class model.

Area of Science:

• Statistics
• Psychometrics
• Social Sciences

Background:

• Rater agreement is crucial for reliable data in various fields.
• Existing agreement models may not fully capture systematic agreement or category quality.
• A need exists for a flexible model to analyze complex rating data.

Purpose of the Study:

• To present a novel latent-class model for quantifying rater agreement.
• To interpret model parameters as systematic agreement and category classification ease.
• To assess the differential quality of rating categories using model fit.

Main Methods:

• Developed a latent-class model where classes represent the "true" category and rater ease of classification.
• Described constrained cases of the model and its relation to existing agreement models (e.g., kappa coefficients).
• Applied the model to empirical data from psychiatric diagnoses and communication studies.

Main Results:

• One model parameter directly estimates the proportion of systematic agreement.
• The model's latent classes reflect the interplay between true category and rater performance.
• Model fit allows for the assessment of differential quality across rating categories.

Conclusions:

• The proposed latent-class model offers a nuanced approach to understanding rater agreement.
• It provides insights into systematic agreement and the quality of rating categories.
• The model is applicable to diverse datasets, including psychiatric diagnoses and behavioral classifications.