Multimodal Sparse Transformer Network for Audio-Visual Speech Recognition.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Speech Processing

Background:

• Automatic speech recognition (ASR) systems face performance degradation in noisy conditions.
• Audio-visual speech recognition (AVSR) uses visual cues to enhance ASR, especially in adverse environments.
• Transformer architectures show promise in AVSR but struggle with irrelevant information and lack motion feature integration.

Purpose of the Study:

• To propose a novel multimodal sparse transformer network (MMST) for enhanced AVSR.
• To address limitations of existing transformer models in handling long-term dependencies and irrelevant information.
• To incorporate essential motion features into AVSR for improved spatio-temporal visual information utilization.

Main Methods:

• Developed a multimodal sparse transformer network (MMST) incorporating sparse self-attention.
• Integrated motion features into the visual modality processing.
• Utilized a cross-modal attention module for seamless information flow between motion and visual modalities.

Main Results:

• The MMST model demonstrated improved attention concentration on relevant global information.
• Integration of motion features enhanced visual feature representation.
• Experiments showed significant performance improvements over state-of-the-art methods on various datasets.
• Reduced word error rate (WER) was achieved, indicating superior recognition accuracy.

Conclusions:

• The proposed MMST effectively enhances audio-visual speech recognition performance, particularly in noisy conditions.
• Incorporating motion features and sparse attention mechanisms are crucial for robust AVSR.
• The MMST offers a promising direction for developing more reliable human-machine interfaces.