Integrated visual transformer and flash attention for lip-to-speech generation GAN

Area of Science:

• Artificial Intelligence
• Speech Technology
• Computer Vision

Background:

• Lip-to-Speech (LTS) generation is rapidly advancing with applications in assistive technology and human-robot interaction.
• Current LTS methods, often GAN-based, struggle with accurate Chinese recognition and aligning lip movements with speech variations.
• Insufficient joint modeling of local and global lip movements leads to visual ambiguities and poor image representation in existing LTS systems.

Purpose of the Study:

• To develop an advanced Lip-to-Speech generation model that overcomes limitations in Chinese recognition and lip movement alignment.
• To enhance the accuracy and quality of synthesized speech from lip movements.
• To improve the practical applicability of LTS technology for communication and accessibility.

Main Methods:

• Proposed FA-GAN (Flash Attention GAN) architecture.
• Separate coding of vision and audio with joint lip motion modeling for improved speech recognition.
• Integration of a multilevel Swin-transformer for enhanced image representation.
• Implementation of a hierarchical iterative generator for superior speech generation.
• Inclusion of a flash attention mechanism to boost computational efficiency.

Main Results:

• FA-GAN demonstrates superior performance on both Chinese and English datasets compared to existing architectures.
• Achieved a Chinese recognition error rate of 43.19%, the lowest reported for this type of model.
• The model effectively addresses challenges in lip-speaking variation and joint lip movement modeling.

Conclusions:

• FA-GAN represents a significant advancement in Lip-to-Speech generation technology.
• The proposed methods enhance speech recognition accuracy, particularly for Chinese.
• This work contributes to improved communication abilities and quality of life for individuals with speech impairments.