Sequential viseme-driven visual speech recognition through dual-stream interactive neural architecture.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Current deep neural network lipreading methods struggle with fine-grained feature extraction, impacting articulatory detail capture.
• Preserving overall semantics often compromises the extraction of crucial local visual features in lipreading models.

Purpose of the Study:

• To introduce a novel paradigm for sentence-level lipreading using sequential viseme knowledge and a dual-stream architecture.
• To address the limitation of fine-grained feature extraction in existing lipreading models.
• To enhance the accuracy and robustness of visual speech recognition systems.

Main Methods:

• Conceptualization of sequential viseme knowledge and development of a dual-stream architecture.
• Integration of sequential viseme dynamics to enhance frame and segment attention.
• Facilitation of interaction between character and viseme granularity information through multiple pathways.

Main Results:

• Achieved state-of-the-art performance on multiple sentence-level lipreading datasets (GRID, CMLR, LRS2, LRS3).
• Demonstrated superior performance with word error rates (WER) as low as 0.6% on GRID and character error rates (CER) of 9.9% on CMLR.
• Showcased robustness to large-scale pretraining and data corruption, highlighting the generalizability of viseme knowledge.

Conclusions:

• The proposed dual-stream architecture effectively addresses fine-grained feature preservation in lipreading.
• Sequential viseme knowledge is crucial for bridging cross-lingual gaps and enhancing visual speech recognition.
• The method demonstrates significant advantages over current top-performing models, with potential for further research and application.