Bayesian Gower agreement for categorical data

Area of Science:

• Statistics
• Data Analysis
• Measurement Science

Background:

• Accurate measurement of agreement is crucial in various fields, including medical research and diagnostics.
• Existing methods for assessing agreement in nominal and ordinal data can be complex or limited in scope.
• Identifying influential units or coders is essential for data quality assessment.

Purpose of the Study:

• To develop and present novel, simple, and intuitive methods for measuring agreement in nominal and ordinal data.
• To extend these methods to accommodate both one-way and two-way random sampling designs.
• To provide a robust approach to Bayesian inference for agreement measures.

Main Methods:

• Utilizing Gower-type distances for calculating agreement scores.
• Developing Bayesian inference approaches for one-way and two-way random sampling designs.
• Applying proposed methods to simulated and real-world datasets, including radiological and psychiatric studies.

Main Results:

• The proposed Gower-distance-based methods are demonstrated to be simple, intuitive, and computationally efficient.
• The methods facilitate the straightforward identification of influential units and/or coders.
• Bayesian inference frameworks are successfully developed for both one-way and two-way designs.
• The study suggests Gaussian mutual information as a potentially more useful agreement scale.

Conclusions:

• The presented methods offer a flexible and accessible approach to quantifying inter-rater agreement.
• The open-source R package 'goweragreement' supports the practical application of these statistical techniques.
• Further research into alternative agreement scales, such as Gaussian mutual information, is warranted.