Automatic assessment of voice quality in the context of multiple annotations

Approaches to evaluate voice quality include perceptual analysis, and acoustical analysis. Perceptual analysis is subjective and depends mostly on the ability of a specialist to assess a pathology, whereas acoustical analysis is objective, but highly relies on the quality of the so called annotations that the specialist assigns to the voice signal. The quality of the annotations for acoustical analysis depends heavily on the expertise and knowledge of the specialist. We face a scenario where we have annotations performed by several specialists with different levels of expertise and knowledge. Traditional pattern recognition methods employed in acoustical analysis are no longer applicable, since these methods are designed for scenarios where a "ground-truth" label is assigned by the specialist. In this paper, we apply recent developments in machine learning for taking into account multiple annotators for acoustical analysis of voice signals. For the classification step we compare two techniques, one of them based on Gaussian Processes for regression with multiple annotators, and the other is a multi-class Logistic Regression model that measures the annotator performance in terms of sensitivity and specificity. The performance of classifiers is assessed in terms of Cohen's Kappa index. Results show that the multi-annotator classification schemes have better performance when compared to techniques based on a traditional classifier where the true label is estimated from the multiple annotations available using majority voting.