Reliability-Based Large-Vocabulary Audio-Visual Speech Recognition

Area of Science:

• Artificial Intelligence
• Computer Vision
• Speech Processing

Background:

• Audio-visual speech recognition (AVSR) enhances performance over audio-only systems for limited vocabularies.
• Current AVSR systems, both hybrid and end-to-end (E2E), underutilize visual information, leading to degraded performance in noisy, large-vocabulary scenarios.

Purpose of the Study:

• To introduce a novel fusion architecture, the Decision Fusion Net (DFN), to optimize the use of multi-modal information in AVSR.
• To improve the robustness and accuracy of large-vocabulary AVSR systems, particularly under adverse acoustic conditions.

Main Methods:

• Developed the Decision Fusion Net (DFN) architecture, incorporating time-variant reliability measures as auxiliary inputs.
• Integrated the DFN into both hybrid and end-to-end (E2E) AVSR frameworks.
• Evaluated the DFN on large-vocabulary datasets: Lip Reading Sentences 2 (LRS2) and 3 (LRS3).

Main Results:

• The DFN significantly improved AVSR performance on large-vocabulary datasets compared to existing systems.
• The hybrid DFN model surpassed the performance of oracle dynamic stream-weighting.
• Achieved relative word error rate reductions of 51% for the hybrid DFN and 43% for the E2E-DFN compared to their respective audio-only baselines.

Conclusions:

• The proposed Decision Fusion Net (DFN) architecture effectively enhances audio-visual speech recognition by optimally fusing audio and visual streams.
• DFN demonstrates superior performance in large-vocabulary settings and noisy conditions, addressing limitations of current AVSR technologies.
• The DFN represents a significant advancement in multi-modal speech recognition research.