Relative observer accuracy for dichotomized variables

We suggest that the relative accuracy of several observers of a disease manifestation can be assessed by comparing the strength of association between their observations and a known cause of that disease. In principle, the observer whose findings are most strongly associated with the causal variable has the least error. In practice many observations appear binary ("normal" or "abnormal") when the underlying distribution of the variable they measure is continuous. For this case of a dichotomized variable, several observers may find different prevalences of abnormality in the same sample either because their accuracy differs or because they use different thresholds above which they regard abnormality as present. A comparison of how strongly their observations are associated with a causal variable using statistical methods for binary observations may lead to incorrect conclusions about the relative accuracy of the observers. We suggest and evaluate a method, based on the biserial correlation coefficient, for comparing the relative accuracy of several observers who may differ in the thresholds they use.