[A mixture model-based rater bias index]

Area of Science:

• Statistics
• Biostatistics
• Psychometrics

Background:

• Rater agreement is crucial for reliable data collection in various fields.
• Basic mixture models provide a framework for analyzing agreement between two observers.
• Existing models may not fully capture complex disagreement patterns.

Purpose of the Study:

• To generalize the basic mixture model for enhanced rater agreement analysis.
• To introduce a novel measure of rater bias using an extended mixture model.
• To improve the nuanced understanding of agreement and disagreement between raters.

Main Methods:

• Utilizing a generalized mixture model with four subpopulations.
• Extending the model to incorporate two latent variables with two classes each.
• Analyzing contingency tables to differentiate agreement and disagreement components.
• Developing a new rater bias index analogous to existing measures.

Main Results:

• The generalized model maintains the core properties of the basic mixture model.
• The enhanced model successfully distinguishes between random agreement and specific disagreement types (upper/lower triangle).
• A new, statistically grounded rater bias measure is proposed.

Conclusions:

• Generalized mixture models offer a more sophisticated approach to rater agreement assessment.
• The proposed model provides a robust framework for identifying sources of disagreement.
• The new rater bias measure offers valuable insights for improving observational consistency.