Multimodal Sensor-Input Architecture with Deep Learning for Audio-Visual Speech Recognition in Wild.

Area of Science:

• Multimodal sensor fusion for artificial intelligence.
• Deep learning applications in signal processing.

Background:

• Audio-visual speech recognition (AVSR) combines audio and visual lip motion data.
• Performance of AVSR degrades in noisy, unconstrained "in the wild" environments.

Purpose of the Study:

• To improve AVSR models for real-world, noisy conditions.
• To investigate the efficacy of data augmentation techniques for AVSR.

Main Methods:

• Implemented deep learning architectures for multimodal sensor fusion.
• Utilized conventional (flips, rotations) and generative adversarial network (GAN) based data augmentation.
• Trained and tested models on LRS2 and LRS3 datasets.

Main Results:

• The proposed AVSR framework with data augmentation demonstrated enhanced performance.
• Augmentation techniques successfully generated more data samples for robust model training.
• Improved recognition accuracy was observed in noisy, in-the-wild scenarios.

Conclusions:

• Data augmentation is a viable strategy to boost AVSR performance in challenging environments.
• The developed AVSR model shows promise for practical applications with unconstrained audio-visual streams.
• Further research can explore advanced augmentation methods for even greater robustness.