Bayesian analysis of paired-comparison sound quality ratings

This paper presents a method to analyze paired-comparison data including either binary or graded ordinal responses, with or without ties. The proposed method can use either of two classical choice models: (1) Thurstone case V, which assumes a Gaussian distribution of the sensory variables underlying listener decisions, or (2) the Bradley-Terry-Luce (BTL) model, which assumes a logistic distribution. The analysis method was validated using simulated paired-comparison experiments with known distributions of the sound-quality parameters in the simulated population from which "participants" were generated at random. The validation indicated that the Thurstone and BTL models give similar results close to the true values. The estimated credibility of a quality difference was slightly higher with the BTL model. The analysis results showed dramatically better precision when the response data included graded ordinal judgments instead of binary responses. Allowing tied responses also tended to improve precision. The method was also applied to data from a real evaluation of hearing-aid programs. The analysis revealed clinically interesting results with high statistical credibility, although the amount of test data was limited.