Log-linear modelling of pairwise interobserver agreement on a categorical scale

Area of Science:

• Statistics
• Biostatistics
• Medical Informatics

Background:

• Accurate assessment of inter-rater reliability is crucial for subjective categorical scales.
• Log-linear models offer a framework for analyzing agreement structures in multidimensional data.
• Previous analyses of pathologist agreement on cervical carcinoma in situ showed variability.

Purpose of the Study:

• To apply log-linear models for describing pairwise agreement among multiple raters.
• To address practical challenges in fitting these models to large, sparse contingency tables.
• To investigate the agreement structure among pathologists evaluating cervical carcinoma in situ.

Main Methods:

• Utilized log-linear models to analyze pairwise agreement.
• Employed the jackknife method for estimating the covariance matrix of model parameters.
• Applied the models to a dataset of seven pathologists classifying cervical carcinoma in situ on a five-level ordinal scale.

Main Results:

• Standard analysis provides consistent parameter estimates but requires adjustments for covariance matrix estimation.
• The jackknife method effectively estimated the covariance matrix.
• Observed near homogeneity in the dependence structure of ratings among the seven pathologists, contrasting with previous findings of differing agreement levels.

Conclusions:

• Log-linear models, with appropriate covariance estimation techniques like the jackknife, can effectively describe complex rater agreement.
• The study demonstrates a consistent underlying structure in how pathologists assess carcinoma in situ, despite potential pairwise differences.
• This methodology provides a robust approach for analyzing inter-rater reliability in medical diagnosis and other fields using subjective scales.