Audio-Visual Fusion Based on Interactive Attention for Person Verification.

Area of Science:

• Computer Science
• Artificial Intelligence
• Biometrics

Background:

• Unimodal person verification systems face performance limitations in complex environments.
• Multimodal feature fusion is crucial for enhancing accuracy and robustness in identity verification.
• Integrating audio-visual information presents challenges in effective feature fusion.

Purpose of the Study:

• To improve multimodal person verification systems by effectively fusing audio and visual features.
• To explore novel fusion models based on attentional mechanisms for enhanced identity verification.
• To evaluate the performance of proposed fusion methods on diverse benchmark datasets.

Main Methods:

• Utilized pretrained models for extracting audio and visual embeddings.
• Developed a baseline fusion model using a fully connected layer.
• Proposed and experimented with three attention-based fusion models: attention, gated, and inter-attention.

Main Results:

• Achieved an Equal Error Rate (EER) of 0.23% and minDCF of 0.011 on VoxCeleb1.
• Obtained an EER of 2.60% and minDCF of 0.283 on NIST SRE19.
• Recorded an EER of 11.30% and minDCF of 0.443 on the CNC-AV dataset.

Conclusions:

• Attention-based fusion models significantly enhance multimodal person verification performance.
• The proposed methods demonstrate superior accuracy and robustness compared to baseline approaches.
• Effective audio-visual feature fusion is key to advancing secure identity verification systems.