The visual sort and rate method for perceptual evaluation in listening tests

This paper introduces the Visual Sort and Rate (VSR) method which can be utilized for perceptual rating of sound stimuli. The method facilitates comparing similar stimuli, thus making the rank ordering of the stimuli easier. To examine the potential benefits of the method, it was compared with two other methods for perceptual rating of audio stimuli. The first method was a straightforward computer-based implementation of a visual analogue scale (VAS) allowing multiple playbacks and re-play of previously heard stimuli (C-VAS). The second method utilized a VAS where the responses were given on paper (P-VAS). The three methods were compared by using two sets of stimuli. The first set was a synthetically generated series of stimuli mimicking the vowel /a/ with different spectral tilts. In this test, a single parameter was rated. The second set of stimuli was a naturally spoken voice. For this set of stimuli three parameters were rated. Results show that the VSR method gave better reliability of the subjects' ratings in the single-parameter tests: Pearson and Spearman correlation coefficients were significantly higher for the VSR method than for the other methods. For the multi-parameter, intra-subject test, significantly higher Pearson correlation coefficients were found for the VSR method than for the VAS on paper.