The effect of anchor modality on the reliability of vocal severity ratings

The purpose of this study was (1) to confirm if anchors (ie, perceptual references) and training affect the inter- and intrarater reliability of perceptual analysis of various voice types and severities compared to receiving training alone, and (2) to determine whether the modality in which the anchor is presented affects rater reliability. In this study, modality refers to whether the anchor is presented auditorily, visually via a written definition (a textual anchor), or a combination of both anchor types. A randomized multigroup comparison was performed. Forty inexperienced judges were selected to rate 36 sustained vowel voice samples of various voice types (ie, normal, breathy, hoarse, and rough) in terms of perceived vocal severity using four different methods (No Anchor, Textual Anchor, Auditory Anchor, and Combined Textual/Auditory Anchors). Subjects were randomly assigned to one of the four conditions. Before the rating task, all subject groups received a brief training sessions (15-20 minutes in duration) in which voice quality type and severity definitions were provided and representative voice samples were listened to. A computer program was developed to present anchors in the form of an auditory sample, written definition, or both. A no anchor condition was also presented. Results indicated that the combination of training and anchors significantly improves the interrater reliability of perceptual voice ratings. In addition, the use of auditory anchors resulted in 95% confidence intervals that were significantly smaller for rating mild voice disorders, and both breathy and hoarse voice qualities. Textual anchors did appear to show some improvement over training alone (ie, no anchors), but were generally not as strong as the use of auditory anchors. However, the combination of textual and auditory anchors resulted in the greatest degree of interrater reliability as assessed via mean correlations. The ratings produced by the Auditory and Combined Anchor groups were also observed to have significantly smaller 95% confidence intervals for the rating of mildly dysphonic voice than the No Anchor or Textual Anchor groups, indicating greater consistency and increased precision of ratings for this level of severity. Anchors were not only useful in improving measures of reliability among judges, but were also able to do so without significantly increasing the amount of time that the judges had to spend on the rating tasks. No significant effect of anchors was observed for measures of intrarater reliability. The use of perceptual anchors (in particular, auditory anchors), in conjunction with training and computerized rating procedures, may provide the consistent methodology necessary to improve the reliability of perceptual judgments. Although textual anchors were not as effective as auditory anchors, the combination of auditory and textual information may result in overall improvements in interrater reliability.