Clinical trials in psychosocial medicine: methodologic and statistical considerations. Part II. Assessing reliability...

The usual statistical procedures for assessing reliability among several raters do not apply to scales of the equal-appearing interval type. This is so primarily because the set of possible responses on such scales, namely the nonnegative integers 0,1,...(M-1), contains so few points that normal distribution theory cannot be invoked. Traditional analyses of such data, which include the one-way analysis of variance intraclass correlation, produce a reliability coefficient based on an assumed normally distributed response variable, and must therefore be interpreted with great caution. This paper discusses a statistical approach which exploits the discrete nature of the response variable and hence is more appropriate for such data.