Speech recognition using an english multimodal corpus with integrated image and depth information

Area of Science:

• Natural Language Processing
• Computer Vision
• Speech Recognition

Background:

• Traditional English corpora often rely on single modalities, limiting information quality and recognition accuracy.
• Existing multimodal corpora may not fully leverage visual and depth information for speech recognition.

Purpose of the Study:

• To develop a method for constructing English multimodal corpora integrating electronic images and depth information.
• To investigate speech recognition methods utilizing this enhanced multimodal corpus.

Main Methods:

• A multimodal fusion strategy combining speech signals with visual information (lip movements, facial expressions) and depth data.
• Application of deep learning for extracting acoustic and visual features.
• Utilizing the Kaldi toolkit's acoustic model for experimental analysis.

Main Results:

• Corpus A, incorporating depth information, demonstrated higher accuracy than Corpus B (lacking depth) across various lip feature dimensions and models.
• At a 10dB signal-to-noise ratio (SNR), Corpus A showed accuracy improvements of 1.4% to 2.6% over Corpus B.
• Specific improvements noted were 2.4% and 1.7% with 15-dimensional lip features, and 1.4% and 2.6% with 32-dimensional lip features.

Conclusions:

• The proposed English multimodal corpus integrating image and depth information achieves high speech recognition accuracy.
• Depth information is a critical factor in improving the accuracy of multimodal speech recognition systems.