User personality and its influence on the performance of pipeline and end-to-end task-oriented dialogue systems

A task-oriented dialogue system with adaptability to user personality could potentially improve dialogue task performance in terms of task success and user satisfaction. In practice, however, the implementation of such a dialogue system would be challenging because of our limited understanding of how user personality influences dialogue task performance. Understanding these influences is essential for developing dialogue strategies that adapt to user personalities, allowing a dialogue system to cater to different users and thereby improve task performance. To examine these influences, we collected data by enrolling crowd-sourced participants to answer personality questionnaires and then chat with a dialogue system to accomplish assigned dialogue tasks. The dialogue tasks were designed using the MultiWOZ dataset, which covers dialogues between a clerk bot and a customer in the context of tourist information access. To clarify the general influence of personality on dialogue task performance across different dialogue systems, we conducted a comparative analysis using both a pipeline system and an end-to-end (E2E) system. We first explored the correlation among user personality, user dialogue behavior, and dialogue task performance through correlation analysis. The results indicate a weak correlation between user personality and dialogue task performance, along with stronger correlations between dialogue behavior with both user personality and dialogue task performance. On the basis of the results of the correlation analysis, we then used Structural Equation Modeling (SEM) to analyze the relationships between user personality and dialogue task performance, using user dialogue behavior as an intermediate variable. The constructed SEM models showed good fit indices. Our results indicate that neutral perceptual sensitivity (which measures a user's sensitivity to external stimuli) and emotional management skill (which measures a user's ability to regulate his/her emotional reactions to situations) have significant influences on his/her dialogue task performance across different dialogue systems. In addition, our findings reveal that personality traits, such as extraversion and conscientiousness, influence dialogue task performance, although such influences may be subject to context-specific variability such as dialogue task design, dialogue domain (MultiWOZ), and system architecture (pipeline).