MVIB-Lip: Multi-View Information Bottleneck for Visual Speech Recognition via Time Series Modeling

Area of Science:

• Computer Science
• Artificial Intelligence
• Signal Processing

Background:

• Lipreading, or visual speech recognition, interprets speech from lip movements.
• Deep learning methods require extensive data and struggle with generalization.
• Existing approaches often fail in low-resource or speaker-independent settings.

Purpose of the Study:

• To develop a more robust and data-efficient lipreading framework.
• To improve generalization capabilities of visual speech recognition systems.
• To explore multi-view learning for integrating complementary lip movement representations.

Main Methods:

• MVIB-Lip framework integrating raw landmark trajectories (multivariate time series) and recurrence plot (RP) images.
• Transformer encoder for temporal sequences and ResNet-18 for RP image features.
• Fusion of two views using product-of-experts posterior and multi-view information bottleneck.

Main Results:

• MVIB-Lip consistently outperformed handcrafted baselines on OuluVS and a self-collected dataset.
• Demonstrated improved generalization for speaker-independent visual speech recognition.
• Validated the effectiveness of combining recurrence plots with deep multi-view learning.

Conclusions:

• Recurrence plots, combined with deep multi-view learning, offer a principled and data-efficient approach for robust visual speech recognition.
• MVIB-Lip presents a promising direction for overcoming limitations of current lipreading techniques.
• The framework shows potential for real-world applications requiring accurate and generalizable lipreading.