Multi-TPC: A Multimodal Dataset for Three-Party Conversations with Speech, Motion, and Gaze

Area of Science:

• Computer Science
• Human-Computer Interaction
• Linguistics

Background:

• Analyzing conversational gestures, particularly in multi-party interactions, is challenging due to limited datasets and evaluation metrics.
• Existing research lacks comprehensive resources for studying the complexities of synchronized speech, motion, and gaze in group conversations.

Purpose of the Study:

• To introduce Multi-TPC, a novel multimodal dataset designed to facilitate the analysis and generation of conversational gestures.
• To provide a foundation for advancing research in discourse analysis, human communication dynamics, and multimodal interaction.

Main Methods:

• Collected and synchronized multimodal data (speech, motion, gaze) from three-party conversations.
• Developed statistical analysis techniques to identify correlations between gestures and other communication modalities.
• Created a dataset and associated models to support research in conversational gesture analysis.

Main Results:

• Established correlations between conversational gestures and audio, text, and speaker identity.
• Demonstrated the utility of the Multi-TPC dataset for studying rich conversational dynamics.
• Highlighted the potential for multimodal analysis in understanding human interaction.

Conclusions:

• The Multi-TPC dataset and associated models offer a significant contribution to the field of multimodal communication research.
• This resource enables deeper investigation into the interplay of gestures, speech, and gaze in group conversations.
• Facilitates future advancements in artificial intelligence systems that require understanding and generating natural human interaction.