Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion.

Area of Science:

• Computer Science
• Signal Processing
• Artificial Intelligence

Background:

• Speaker diarization aims to segment and label speech segments by speaker.
• Existing methods struggle with multi-participant interactions, especially when participants are not facing the camera/microphone.
• Challenges include tracking moving individuals and associating speech with the correct person.

Purpose of the Study:

• To propose a novel audio-visual spatiotemporal diarization model.
• To address the limitations of current diarization systems in complex, dynamic multi-party conversations.
• To improve the accuracy of speaker identification and speech turn segmentation in challenging scenarios.

Main Methods:

• Combines multiple-person visual tracking with multiple speech-source localization.
• Employs a novel audio-visual fusion technique involving binaural spectral feature extraction, supervised audio-visual alignment, and semi-supervised clustering.
• Utilizes a latent-variable temporal graphical model for inferring speaker identities and speech turns.

Main Results:

• The proposed method effectively handles simultaneous speech from multiple speakers.
• Achieves accurate speech-to-person association even with moving participants and head turns.
• Demonstrates efficient exact inference and robust performance compared to state-of-the-art algorithms.

Conclusions:

• The developed audio-visual diarization model offers a principled approach to complex dialogue scenarios.
• The novel fusion and inference methods significantly advance speaker diarization capabilities.
• Introduces a valuable new dataset for training and evaluating audio-visual diarization systems.