Speech Driven Gaze in a Face-to-Face Interaction

Area of Science:

• Multimodal Communication
• Human-Computer Interaction
• Computational Linguistics

Background:

• Gaze and language are key components of human interaction.
• Gaze, a non-verbal cue, provides vital social signals but is less studied than language.
• Understanding gaze-speech dynamics is crucial for developing sophisticated communication models.

Purpose of the Study:

• Investigate gaze direction (aversion vs. face gaze) and its relationship with speech in face-to-face interactions.
• Develop and evaluate a computational model for multimodal communication that predicts gaze direction from speech features.
• Analyze differences in gaze patterns based on participant roles in a controlled setting.

Main Methods:

• Collected eye-tracking and speech data from 28 pairs of participants in a mock job interview.
• Annotated speech data using the ISO 24617-2 Standard for Dialogue Act Annotation and social gaze tags.
• Employed Convolutional Neural Network (CNN) architectures, VGGNet and ResNet, for comparative analysis and gaze prediction.

Main Results:

• Significant differences in gaze frequency and duration were observed based on participant roles.
• The ResNet model achieved over 70% accuracy in predicting gaze direction.
• Established a correlation between specific speech features and gaze behavior.

Conclusions:

• Gaze patterns are significantly influenced by conversational roles.
• Computational models, particularly ResNet, show promise in predicting gaze direction from speech.
• This research contributes to a deeper understanding of multimodal communication and informs the development of AI systems.