A variance-based measure of inter-rater agreement in medical databases

The increasing use of encoded medical data requires flexible tools for data quality assessment. Existing methods are not always adequate, and this paper proposes a new metric for inter-rater agreement of aggregated diagnostic data. The metric, which is applicable in prospective as well as retrospective coding studies, quantifies the variability in the coding scheme, and the variation can be differentiated in categories and in coders. Five alternative definitions were compared in a set of simulated coding situations and in the context of mortality statistics. Two of them were more effective, and the choice between them must be made according to the situation. The metric is more powerful for larger numbers of coded cases, and Type I errors are frequent when coding situations include different numbers of cases. We also show that it is difficult to interpret the meaning of variation when the structures of the compared coding schemes differ.