Spatial-temporal activity-informed diarization and separation.

Area of Science:

• Signal Processing
• Machine Learning
• Acoustics

Background:

• Accurate speaker diarization and separation are crucial for many audio processing tasks.
• Existing methods often require prior knowledge of microphone array configurations or suffer from high computational complexity.

Purpose of the Study:

• To propose a robust multichannel speaker diarization and separation system.
• To develop a hybrid architecture combining array signal processing and deep learning.
• To achieve high performance with low computational complexity.

Main Methods:

• A hybrid system integrating array signal processing and deep learning units.
• Computation of a spatial coherence matrix using whitened Relative Transfer Functions for speaker diarization.
• Development of an Encoder-Decoder-based Attractor network for speaker activity estimation.
• Proposal of a Global and Local Activity-driven Speaker Extraction network for speaker separation.

Main Results:

• The proposed system demonstrates superior performance in speaker diarization, counting, and separation.
• The system achieves this performance with significantly lower computational complexity compared to baseline methods.
• The spatial coherence matrix serves as a robust feature, independent of array configuration.

Conclusions:

• The hybrid approach effectively exploits spatiotemporal speaker activity for robust diarization and separation.
• The developed system offers a computationally efficient and high-performing solution for multichannel audio processing.
• This work advances the state-of-the-art in speaker diarization and separation systems.