Psychoacoustic ranking and selection using modified knockout tournaments

This paper introduces a ranking and selection approach to psychoacoustic and psychophysical experimentation, with the aim of identifying top-ranking samples in listening experiments with minimal pairwise comparisons. We draw inspiration from sports tournament designs and propose to adopt modified knockout (KO) tournaments. Two variants of modified KO tournaments are described, which adapt the tree selection sorting algorithm and the replacement selection algorithm known from computer science. To validate the proposed method, a listening experiment is conducted, where binaural renderings of seven chamber music halls are compared regarding loudness and reverberance. The rankings obtained by the modified KO tournament method are compared to those obtained from a traditional round-robin (RR) design, where all possible pairs are compared. Moreover, the paper presents simulations to illustrate the method's robustness when choosing different parameters and assuming different underlying data distributions. The study's findings demonstrate that modified KO tournaments are more efficient than full RR designs in terms of the number of comparisons required for identifying the top ranking samples. Thus, they provide a promising alternative for this task. We offer an open-source implementation so that researchers can easily integrate KO designs into their studies.