[Evaluation of the structures of agreement and disagreement in reliability studies]

The kappa coefficient has been the measurement preferred by epidemiologists for reliability studies. Various articles have demonstrated that the use of the kappa coefficient may have some undesirable features in certain contexts. Recently, methodologies using an ordinal scale for the modelling of interobserver agreement have been developed as an alternative to kappa. To show that there is a class of log-linear statistical models that when analyzed sequentially can be used to rate the patterns of agreement and disagreement. Using data on the comparability of primary and proxy respondent reports with respect to the frequency of alcoholic consumption and its correlation to coronary diseases a nested set of log-linear models was adjusted to find the "best" model. Computed odds ratios to determine the measure of agreement were also computed. The weight kappa was equal 0.685 with 95% CI (0.638-0.732) showing a good agreement. But it does not give any information about the structure of the agreement and disagreement. Among the sequence of models analyzed, the one with the best adjustment showed an agreement estimated at 0.4454 with 95% CI (0.1300-0.7608) and an association estimated at 1.3309 with 95% CI (0.9649-1.6978). The measure tau for adjacent categories was 9.2 with 95% CI (6.0-14.2). Thus, evidence shows that the observers tended to rate many phenomena similarly. Furthermore, high (or low) ratings made by primary respondents tended to be associated with high (or low) ratings made by the proxy respondents. Log-linear models can give us a more informative and more complete analysis with respect to the rating of matched pairs of observers than that given by kappa. In conclusion, the indiscriminate use of kappa as the only agreement index must be questioned. The appendix demonstrates how to use PROC GENMOD in SAS to fit these models.