The Co-Structuring of Gesture-Vocal Dynamics: An Exploration in Karnatak Music Performance

Area of Science:

• Music Performance Studies
• Biomusicology
• Computational Auditory Processing

Background:

• Vocalists often use hand and body movements while singing.
• The relationship between vocal phrases and gestures is not well understood.
• Individual differences in singers' gesturing styles require investigation.

Purpose of the Study:

• To develop a quantitative pipeline for analyzing vocal-gesture co-structuring.
• To characterize the multidimensional interplay between body movements and vocalizations.
• To explore individual differences in multimodal performance styles.

Main Methods:

• Applied a quantitative analysis and visualization pipeline.
• Utilized audio and motion tracking data from 44 Karnatak music performances.
• Analyzed time-varying features of vocalizations and gestures.

Main Results:

• Vocal and gesture features showed higher similarity when concurrent vocal features were also similar.
• Each vocalist exhibited unique dimensions (e.g., hand position, head acceleration) for movement-singing co-structuring.
• Co-structuring reduced the overall degrees of freedom in multimodal performance.

Conclusions:

• A computational approach can characterize individual vocalists' unique gesture-vocal co-structuring profiles.
• Similar-sounding musical motifs tend to co-structure with similar gestures.
• The method is applicable to diverse multimodal communication systems.