Improving ultrasound-based multimodal speech recognition with predictive features from representation learning

Area of Science:

• Speech Technology
• Machine Learning
• Biomedical Imaging

Background:

• Representation learning aims to extract high-level features from temporal data.
• Accurate speech recognition relies on effective feature extraction from articulatory signals.
• Previous methods often struggle to capture complex dynamics in speech production.

Purpose of the Study:

• To evaluate the effectiveness of representation learning for speech recognition using articulatory data.
• To compare predictive features from a 3D CNN with traditional methods like auto-encoders and DCT.
• To assess the impact of combining tongue and lip imaging modalities on speech recognition performance.

Main Methods:

• A 3D convolutional neural network (CNN) was trained to predict future frames from ultrasound tongue and optical lip images.
• Extracted features were used in a continuous hidden Markov model (HMM) based speech recognition system.
• Performance was evaluated using word error rates (WER) against baseline methods and different acoustic models.

Main Results:

• Predictive tongue features significantly reduced word error rates compared to auto-encoder or DCT features.
• Improvements were observed for both Gaussian mixture model (GMM) and deep neural network (DNN) acoustic models.
• Combining tongue and lip modalities showed a reduced advantage for predictive features compared to single-modality use.

Conclusions:

• Representation learning, particularly using predictive features from ultrasound tongue imaging, enhances speech recognition.
• Predictive features offer a superior alternative to traditional feature extraction methods for articulatory speech data.
• Future research should explore optimal fusion strategies for multi-modal articulatory data in speech recognition.