Multimodal indices to Japanese and French prosodically expressed social affects

Area of Science:

• Psychology
• Linguistics
• Communication Studies

Background:

• Limited research exists on the audiovisual production of controlled social affects, such as attitudinal expressions.
• Understanding the interplay of audio and visual channels in affective communication is crucial.

Purpose of the Study:

• To investigate the contribution of audio and visual modalities in the perception of attitudinal expressions.
• To analyze the relationship between objective acoustic/visual parameters and perceived social affects.
• To explore bilingual differences in the expression and perception of these affects.

Main Methods:

• A perception study involving audiovisual expressions of 12 Japanese and 6 French attitudes.
• Analysis of the relative importance of audio and visual modalities in decoding expressions from four speakers.
• Acoustic (F0, duration, intensity) and visual (Action Units) analysis of two speakers' productions.
• Bilingual comparison of objective features and their perceptual relevance.

Main Results:

• Identified the relative importance of audio and visual modalities in the perceptual decoding of attitudinal expressions.
• Matched objective acoustic and visual parameters with listener perception of social affects.
• Highlighted language-specific features, such as the role of fundamental frequency and voice quality in Japanese attitudinal expressions.

Conclusions:

• Both audio and visual channels significantly contribute to affective communication, with varying importance depending on the specific attitude and language.
• Objective acoustic and visual features can be linked to the perception of social affects.
• Cross-linguistic analysis reveals distinct patterns in the expression of social affects, emphasizing the need for a bilingual perspective.