Statistical techniques for comparing measurers and methods of measurement: a critical review

1. Clinical and experimental pharmacologists and physiologists often wish to compare two methods of measurement, or two measurers. 2. Biostatisticians insist that what should be sought is not agreement between methods or measurers, but disagreement or bias. 3. If measurements have been made on a continuous scale, the main choice is between the Altman-Bland method of differences and least products regression analysis. It is argued that although the former is relatively simple to execute, it does not distinguish adequately between fixed and proportional bias. Least products regression analysis, although more difficult to execute, does achieve this goal. There is almost universal agreement among biostatisticians that the Pearson product-moment correlation coefficient (r) is valueless as a test for bias. 4. If measurements have been made on a categorical scale, unordered or ordered, the most popular method of analysis is to use the kappa statistic. If the categories are unordered, the unweighted kappa statistic (K) is appropriate. If the categories are ordered, as they are in most rating scales in clinical, psychological and epidemiological research, the weighted kappa statistic (K(w)) is preferable. But K(w) corresponds to the intraclass correlation coefficient, which, like r for continuous variables, is incapable of detecting bias. Simple techniques for detecting bias in the case of ordered categorical variables are described and commended to investigators.