A simple approach to interrater agreement

We have described a simple method for computing a measure of interrater agreement or reliability. This method depends on an experimental design that uses a consensus rating as well as individual ratings. The computations required are counting and percentages; the overall agreement index uses a simple weighting average.