Generalizability Theory II: Application to perceptual scaling of speech naturalness in adults who stutter

Generalizability theory has been recommended as the most comprehensive method for assessing the reliability of observational data. It provides a framework for calculating the various sources of measurement error and allows further design of measurements for a particular purpose. This paper gives a practical illustration of how this method may be used in the analysis of observational data. We use the ratings of 15 unsophisticated raters using the 9-point speech naturalness scale of R. R. Martin, S. K. Haroldson, and K. A. Triden (1984) to evaluate the speech of adults before and after treatment for stuttering. We calculate various sources of measurement error and use these to estimate the minimum number of raters and ratings per rater for a reliable result. For posttreatment data, the average of three independent raters, and for pretreatment data, the average of five independent raters should give a result within one scale point of the hypothetical true score for the speaker in at least 80% of samples. The example illustrates the advantages of using this method of analysis.