Multimodal Speaker Diarization Using a Pre-Trained Audio-Visual Synchronization Model.

Area of Science:

• Computer Science
• Signal Processing
• Artificial Intelligence

Background:

• Speaker diarization systems identify speakers in recordings.
• Existing methods often rely solely on audio data.
• Multimodal approaches offer potential for improved accuracy.

Purpose of the Study:

• To propose a novel multimodal speaker diarization technique.
• To leverage audio-visual synchronization for identifying active speakers.
• To enhance the accuracy of speaker diarization systems.

Main Methods:

• Utilized a pre-trained audio-visual synchronization model.
• Employed face detection to extract face-only video segments.
• Matched audio frames with synchronized visual input using a two-streamed network.
• Trained Gaussian Mixture Model (GMM)-based clusters on high-confidence segments.

Main Results:

• Achieved significant improvement in Diarization Error Rate (DER) compared to audio-only methods.
• Demonstrated performance close to state-of-the-art multimodal diarization techniques.
• Validated the effectiveness on the AMI meeting corpus.

Conclusions:

• The proposed multimodal approach offers a simple yet effective solution for speaker diarization.
• Audio-visual synchronization is a valuable component for accurate speaker identification.
• This method provides a strong baseline for future multimodal diarization research.