What you need to know about statistics, part II: reliability of diagnostic and screening tests

The foundation for the usefulness of any diagnostic test should be that it is both reliable and accurate in its clinical diagnosis. In this article we present the second of a two-part series on validity and reliability, discussing the assessment of reliability among raters of diagnostic tests and between diagnostics tests themselves. To examine reproducibility (reliability) among raters of diagnostic tests we present the calculation of two statistical procedures: (1) the kappa coefficient statistic when presented with categorical data for the presence or absence of a clinical diagnosis and (2) the intraclass correlation coefficient (ICC) for continuously scaled data among raters. The accuracy among diagnostic tests (i.e. their interchangeability) can be evaluated by application of (1) a Bland-Altman plot procedure (with its 95% limits of agreement) and (2) the Passing-Bablok regression procedure (for the identification and evaluation of systematic and proportional differences). When deciding whether to select a diagnostic test one must evaluate its ability to provide more precise information than a gold standard test, and whether in clinical practice it would be more beneficial for patients to adopt it.