Multisensory Fusion for Unsupervised Spatiotemporal Speaker Diarization.

Area of Science:

• Speech processing
• Machine learning
• Acoustic signal processing

Background:

• Speaker diarization identifies speakers in audio recordings.
• Spatial features from microphone arrays can aid speaker identification in meetings.
• Current methods often rely solely on audio features, potentially missing spatial cues.

Purpose of the Study:

• To develop and evaluate a framework combining speaker embeddings with Time Difference of Arrival (TDOA) for improved speaker diarization.
• To assess the effectiveness of different speaker embedding models (ECAPA-TDNN, X-vectors) when integrated with spatial information.
• To determine optimal methods for combining spatial-temporal information for clustering speakers.

Main Methods:

• Speaker embeddings extracted using ECAPA-TDNN and X-vectors.
• Time Difference of Arrival (TDOA) calculated using Generalized Cross-Correlation (GCC) with Phase Transform (PHAT) weighting.
• Framework evaluated on AVLab and SpeaD-M3C multichannel datasets, assessing various spatial-temporal combination techniques.

Main Results:

• Integration of spatial information significantly improves state-of-the-art deep learning diarization models.
• A 2-3% reduction in Diarization Error Rate (DER) was observed compared to baseline approaches.
• ECAPA-TDNN generally outperformed X-vectors, but both benefited from spatial data integration.

Conclusions:

• Combining speaker embeddings with spatial TDOA data is an effective strategy for enhancing speaker diarization performance.
• Spatial information provides valuable cues that complement audio-based speaker recognition, especially in meeting environments.
• The proposed framework demonstrates the utility of leveraging multichannel audio for more accurate and robust speaker identification.