An Effective Conversion of Visemes to Words for High-Performance Automatic Lipreading

Area of Science:

• Computer Science
• Artificial Intelligence
• Signal Processing

Background:

• Viseme-based lipreading systems show promise for sentence decoding.
• System performance is bottlenecked by inefficient viseme-to-word conversion.
• Homophenes, where visemes map to multiple words, cause significant accuracy drops.

Purpose of the Study:

• To develop an efficient deep learning model for viseme-to-word conversion.
• To address the challenge of homophenes in lipreading.
• To improve the accuracy of spoken sentence prediction from videos.

Main Methods:

• A deep learning network model utilizing an Attention-based Gated Recurrent Unit (AGRU) was proposed.
• The AGRU model was compared against three alternative approaches.
• Experiments were conducted on the LRS2 and LRS3 benchmark datasets.

Main Results:

• The proposed AGRU model demonstrated robustness, high efficiency, and short execution times.
• The model effectively converted visemes to words and discriminated between homophenes.
• Achieved a word accuracy rate of 79.6% on the LRS2 dataset, a 15.0% improvement over state-of-the-art.

Conclusions:

• The developed AGRU model significantly enhances viseme-to-word conversion accuracy in lipreading.
• The model's efficiency and robustness make it suitable for practical applications.
• This approach offers a substantial improvement for predicting spoken sentences from visual data.