Efficiency considerations in the analysis of inter-observer agreement.

Area of Science:

• Biostatistics
• Psychometrics
• Medical Informatics

Background:

• Inter-rater and intra-rater reliability are crucial for the consistency of binary assessments.
• The kappa statistic is a common measure, but its application can be limited.
• Existing models may not fully leverage replicate measurements for enhanced reliability estimation.

Purpose of the Study:

• To develop a statistical model for estimating inter-rater and intra-rater reliability using replicate measurements.
• To evaluate the impact of obtaining two measurements per subject on reliability estimation efficiency.
• To compare the precision gains from replicate measurements in binary assessment studies.

Main Methods:

• Development of a nested beta-binomial model analogous to Rosner's (1989, 1992) framework.
• Application of the model to scenarios with one or two replicate measurements per subject.
• Analysis of the efficiency and precision of reliability estimates based on the number of measurements.

Main Results:

• The proposed model effectively estimates both inter-rater and intra-rater reliability with replicate data.
• Increasing measurements per rater from one to two enhances the precision of reliability estimates.
• Fewer subjects may be required when using two measurements per rater without compromising inter-rater reliability estimation efficiency.

Conclusions:

• Replicate measurements in binary assessments offer a viable strategy to improve reliability estimation.
• The developed model provides a robust method for analyzing data with replicate observations.
• This approach can lead to more efficient study designs in fields relying on consistent binary assessments.