Semi-supervised audio-driven TV-news speaker diarization using deep neural embeddings.

Area of Science:

• Computer Science
• Artificial Intelligence
• Multimedia Analysis

Background:

• Speaker diarization is crucial for understanding multimedia content.
• Existing methods often struggle with complex audio-visual data.
• Accurate speaker identification enhances content accessibility and analysis.

Purpose of the Study:

• To introduce and evaluate an audio-driven, multimodal approach for speaker diarization.
• To leverage deep learning for generating audio-visual embeddings.
• To improve speaker diarization performance through a novel fusion stage.

Main Methods:

• Semi-supervised clustering of audio-visual embeddings.
• Utilizing Long Short-Term Memory (LSTM) Siamese networks for audio embeddings.
• Employing pre-trained Convolutional Neural Networks (CNNs) for video-based facial embeddings.
• Implementing a fusion stage combining audio and video information.

Main Results:

• The proposed multimodal approach demonstrates improved speaker diarization performance.
• Evaluation on large-scale datasets (VoxCeleb, AVL-SD) validates the method's effectiveness.
• The system successfully handles peculiarities of TV news content.

Conclusions:

• The developed audio-visual speaker diarization method offers a significant advancement.
• The open-source release promotes reproducible research and collaboration.
• This approach enhances the analysis of multimedia content by accurately identifying speakers.