Learning the Relative Dynamic Features for Word-Level Lipreading.

Area of Science:

• Computer Vision
• Speech Recognition
• Biomedical Engineering

Background:

• Lipreading, or speechreading, analyzes lip movements for speech recognition but faces challenges like homophones and speaker variability.
• Existing methods struggle with capturing the complex spatial-temporal dynamics of lip motion.
• Accurate lipreading is crucial for applications in noisy environments and for individuals with hearing impairments.

Purpose of the Study:

• To develop an efficient two-stream model for word-level lipreading.
• To improve the extraction of spatial-temporal features from lip movements.
• To achieve state-of-the-art performance in lipreading tasks.

Main Methods:

• Proposed an efficient two-stream Convolutional Neural Network (CNN) model.
• Utilized separate streams for extracting static (single-frame) and dynamic (multi-frame) lip movement features.
• Investigated optimized convolution structures, sampling methods for different channels, and fusion techniques for model components.

Main Results:

• Achieved an 8% improvement by exploring effective convolution structures.
• Demonstrated the impact of different sampling methods on fast and slow channels.
• Showcased the influence of various fusion methods on the two-stream network's performance.

Conclusions:

• The proposed two-stream model significantly enhances word-level lipreading accuracy.
• The model's ability to capture spatial-temporal dynamics addresses key challenges in lipreading.
• Achieved new state-of-the-art results on two large-scale lipreading datasets, validating the approach.