Reliability of speech naturalness ratings of stuttered speech during treatment

This study evaluated the reliability with which relatively sophisticated and unsophisticated judges used a 9-point scale to rate the speech naturalness of speech samples from 10 clients in a treatment program for stuttering that employed prolonged speech. Judges rated repeated speech samples from different speakers during various phases of the program. Different groups of sophisticated and unsophisticated judges made ratings at either 15 sec, 30 sec, or 60 sec intervals while listening to the samples. Of the reliability indices, intraclass correlations were significantly higher for sophisticated judges although the consistency and agreement of unsophisticated judges were generally equivalent to that of sophisticated judges. Both agreement scores and intraclass correlations were higher when ratings were made at 60 sec rather than 30 sec intervals. The predominant variable that influenced judgement reliability appeared to be differences among the subjects. The methodology partially replicated Martin, Haroldson, and Triden's (1984) initial investigation on the use of this scale. However, the levels of intra- or interjudge reliability in this study were lower than the levels achieved by Martin et al.'s judges. There were important differences between the Martin et al. study and this one that may account for the findings, and these are discussed.