Using replicate observations in observer agreement studies with binary assessments

By introducing replicate observations into observer agreement studies, one can obtain better measures of observer agreement than heretofore possible. New methodology based on the analysis of latent variables allows a separation of within- and between-observer variation for binary measures of assessment among pairs of observers. Maximum likelihood estimation and hypothesis testing are discussed. The methodology is illustrated using data on the assessment of dysplasia by pathologists.