A comparison model for measuring individual agreement.

Area of Science:

• Biostatistics
• Data Analysis
• Scientific Measurement

Background:

• Assessing inter-rater and intra-rater reliability is crucial in scientific studies.
• Existing methods may not adequately compare agreement across multiple raters and replicated readings.
• A flexible model is needed to evaluate individual agreement comprehensively.

Purpose of the Study:

• To propose a general comparison model for assessing individual agreement among multiple raters with replicated readings.
• To introduce two novel comparative agreement indices: Total-Intra Ratio (TIR) and Intra-Intra Ratio (IIR).
• To provide a framework for exploring total-rater versus intrarater agreement and comparing intrarater agreement among raters.

Main Methods:

• Development of a general comparison model based on mean squared deviations (MSDs).
• Introduction of TIR for noninferiority assessment of inter-rater agreement relative to intrarater agreement.
• Introduction of IIR for classical assessment of rater precision.
• Utilizing Generalized Estimating Equations (GEE) methodology for estimation and statistical inference.

Main Results:

• The proposed model allows flexible comparison of total-rater and intrarater agreement.
• TIR provides a noninferiority assessment applicable with or without a reference standard.
• IIR enables direct comparison of precision among selected raters.
• The FDA's bioequivalence method is shown as a special case of the TIR approach.

Conclusions:

• The proposed model and indices offer a robust framework for evaluating individual rater agreement in studies with multiple raters and readings.
• TIR and IIR provide valuable metrics for ensuring data reliability and understanding rater performance.
• The GEE methodology ensures sound statistical inference for the proposed indices.