Interobserver reliability and perceptual ratings: more than meets the ear

The purpose of this study was to examine the reliability of ratings of perceptual characteristics for 10 ataxic dysarthric subjects. The influence of the occurrence of "deviant" speech parameters on the calculation of reliability coefficients was also explored. Results indicated that overall interobserver agreement levels for minimally trained judges compared favorably to reliability coefficients reported in previous studies. Furthermore, levels of overall agreement were above levels of agreement expected on the basis of chance alone. In contrast to overall interobserver agreement, much lower levels of interobserver agreement were obtained when "occurrence reliability" coefficients were calculated for deviant dimensions alone. However, occurrence reliability coefficients surpassed the level of agreement expected on the basis of chance alone for all subjects. Based on the results of this investigation, recommendations are made for modifying standard practices for obtaining interobserver reliability for perceptual ratings of speech characteristics.