Automated classification of conversation valence and arousal using autonomic nervous system responses

The quality of dyadic cooperation or conversation can be predicted from the interacting individuals' physiological responses, and several studies have combined physiological responses with machine learning algorithms to classify conversation states. However, most of these studies have focused on either a single physiological response or two-class classification. In this study, we used multiple autonomic nervous system responses and pattern recognition algorithms to automatically classify four dyadic conversation scenarios corresponding to four quadrants of the arousal-valence space.

Heart rate, electrodermal activity, respiration, and peripheral skin temperature were measured from N = 51 dyads in all four scenarios, and audio/video of the dyads were simultaneously recorded. Physiological data were classified into the four scenarios using multiple different feature selection and classification algorithms. For comparison, audio/video was classified manually into the four scenarios by human raters, and dyads' self-report questionnaire answers were classified into the four scenarios using classification algorithms as well.

The highest 4-class classification accuracy achieved using physiological responses was 51.5%. Conversely, human raters achieved an accuracy of 69.6% based on manual review of audio/video, and an accuracy of 79.5% was achieved using participants' self-report questionnaire responses.

While the study is considered exploratory since many algorithms were tested, we conclude that dyadic autonomic nervous system responses can be used to classify conversation valence and arousal, but such physiological responses are not as accurate as human observers or self-report questionnaires. We finally propose potential additional analyses as well as three directions for future experiments: combining physiological responses with other data types, scenarios where participants have different perceptions of the interaction, and clinical populations.