From tetrachoric to kappa: How to assess reliability on binary scales.

Area of Science:

• Psychometrics
• Statistical Methods

Background:

• Reliability in psychometrics is vital for measurement accuracy.
• Binary outcomes pose unique challenges for reliability estimation compared to quantitative scales.
• Classical test theory and intraclass correlation coefficients are established for quantitative data.

Purpose of the Study:

• To review and link three major approaches for estimating reliability of single ratings on binary scales.
• To clarify conceptual relationships and evaluate performance across repeatability and reproducibility studies.
• To extend Bayesian methods for kappa coefficients and estimate manifest scale reliability.

Main Methods:

• Comparison of normal approximation, kappa coefficients, and latent variable approaches.
• Extension of Bayesian Dirichlet-multinomial method for multi-replicate kappa coefficients.
• Introduction of a Bayesian method for manifest scale reliability estimation from latent scale reliability.

Main Results:

• The normal approximation approach demonstrated poor performance.
• The frequentist approach exhibited unreliability due to singularity issues.
• The latent variable approach provides a robust framework for binary scale reliability.

Conclusions:

• The normal approximation and frequentist methods are not recommended for binary scale reliability.
• The latent variable approach offers a more reliable method for assessing binary outcome reliability.
• Refined practical recommendations are provided for reliability estimation in psychometric research.