Data stream-pairwise bottleneck transformer for engagement estimation from video conversation

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Modeling participant engagement in multiparty conversations requires effective handling of multimodal data streams (video, audio).
• Previous methods using global token-based transformers faced challenges with redundancy in participant-feature estimation across modalities and participants.

Purpose of the Study:

• To develop and evaluate a novel joint model for assessing participant engagement in multiparty conversations.
• To address the redundancy issue in standard cross-attention transformers for multimodal interaction modeling.

Main Methods:

• A joint model utilizing global token-based transformers was proposed, treating all data streams (video, audio) without distinguishing cross-modal or cross-participant interactions.
• The model was experimented on the RoomReader corpus to assess its performance.

Main Results:

• The proposed joint model demonstrated superior performance compared to previous approaches.
• Achieved accuracy scores ranging from 0.720 to 0.763.
• Weighted F1 scores ranged from 0.733 to 0.771, and macro F1 scores ranged from 0.236 to 0.277.

Conclusions:

• The developed joint model effectively models interactions among all data streams for assessing participant engagement.
• The findings suggest that a unified approach to multimodal interaction modeling can improve performance in analyzing complex conversational dynamics.