Sequential viseme-driven visual speech recognition through dual-stream interactive neural architecture

While achieving considerable success as a sequence-to-sequence prediction task, current deep neural network-based sentence-level lipreading methods exhibit a fundamental limitation: the preservation of overall semantics often comes at the expense of local fine-grained visual feature extraction, potentially compromising the model's ability to capture crucial articulatory details. This study introduces a novel paradigm through the conceptualization of sequential viseme knowledge and the development of an innovative dual-stream architecture. By integrating sequential viseme dynamics, the framework enhances localized frame and segment attention within the sequence prediction task, effectively addressing the fine-grained feature preservation issue while maintaining temporal coherence. Meanwhile, the dual-stream architecture fully harnesses the potential for viseme prediction, facilitating interaction between coarse (character) and fine (viseme) granularity information from multiple pathways, thus jointly enhancing the performance of lipreading. We conduct meticulously designed sequential viseme annotation and comprehensive experiments on multiple classic sentence-level lipreading datasets encompassing both Chinese and English. Our method demonstrates superior performance by either surpassing existing state-of-the-art results or achieving competitive performance across various benchmarks. With a word error rate (WER) of 0.6 % on GRID and a character error rate (CER) of 9.9 % on CMLR, our method delivers the best known performance under the condition of no external data or pretrained knowledge. For the LRS2 and LRS3 datasets, under the condition of equivalent 818 hours of training data, our method achieves state-of-the-art performance, with WER of 23.3 % and 28.9 %, respectively. These results demonstrate the robustness of our method to large-scale pretraining and further highlight its distinctive advantages compared to current top-performing models with more non-publicly available training data. Through systematic experimental validation, we reveal the remarkable generalizability of viseme knowledge in challenging scenarios and substantiate the critical role of our dual-stream architecture in enhancing model robustness against data corruption. Furthermore, we elucidate the fundamental mechanisms and intrinsic interpretability of viseme knowledge, which serves as a critical element for bridging cross-lingual gaps and enhancing overall performance in visual speech recognition. We will publicly release our annotated sequential viseme data and maintain the implementation code in a dedicated GitHub repository (https://github.com/clayh24/lipreading-based-on-sequential-viseme), to spur further research in this field.