Lip Reading by Alternating between Spatiotemporal and Spatial Convolutions.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Lip reading (LR) involves predicting speech from visual cues.
• Effective feature learning from visual speech sequences is crucial for LR performance.
• Integrating spatiotemporal and spatial information processing is key to enhancing LR systems.

Purpose of the Study:

• To investigate the benefits of alternating spatiotemporal and spatial convolutions for lip reading.
• To introduce and evaluate a novel learnable module, ALSOS, for lip reading.
• To enhance the accuracy of end-to-end trained lip reading systems.

Main Methods:

• A novel module, ALSOS (Alternating Spatiotemporal and Spatial Convolutions), was developed.
• ALSOS integrates 3D (spatiotemporal) and 2D (spatial) convolutions with conversion components for sequence-to-sequence mapping.
• The proposed system incorporates ALSOS within ResNet blocks and uses Temporal Convolutional Networks (TCNs) for classification, trained end-to-end on image sequences.

Main Results:

• The ALSOS module demonstrated improved performance in lip reading tasks.
• Experiments on Greek and English datasets (LRW-500) showed enhanced classification accuracy.
• Integrating ALSOS with ResNet architecture improved performance by capturing temporal information at various spatial scales.

Conclusions:

• The ALSOS module effectively captures spatiotemporal dynamics beneficial for lip reading.
• The proposed ALSOS module offers an advantageous enhancement to existing ResNet-based lip reading systems.
• Alternating spatiotemporal and spatial convolutions provides a valuable approach for improving visual speech recognition.