Prompt Tuning of Deep Neural Networks for Speaker-Adaptive Visual Speech Recognition

Area of Science:

• Artificial Intelligence
• Computer Vision
• Speech Processing

Background:

• Visual Speech Recognition (VSR) infers speech from lip movements.
• VSR models struggle with unseen speakers due to variations in lip appearance and movement.
• Adapting VSR models to new speakers is crucial for broader applicability.

Purpose of the Study:

• To develop speaker-adaptive Visual Speech Recognition using prompt tuning methods.
• To improve the performance of pre-trained VSR models on unseen speakers.
• To investigate the effectiveness of different prompt types for VSR adaptation.

Main Methods:

• Proposed prompt tuning methods for Deep Neural Networks (DNNs) in speaker-adaptive VSR.
• Explored addition, padding, and concatenation prompt forms applicable to CNN and Transformer architectures.
• Finetuned prompts on small adaptation datasets for target speakers, avoiding modification of pre-trained model parameters.

Main Results:

• Significant performance improvement of VSR models on unseen speakers using minimal adaptation data (under 5 minutes).
• Prompt tuning effectively adapted pre-trained VSR models despite existing speaker variations.
• Analysis provided insights into prompt tuning's advantages over traditional finetuning.

Conclusions:

• Prompt tuning offers an efficient approach for speaker adaptation in VSR.
• The proposed methods demonstrate robustness and effectiveness across different VSR tasks and datasets.
• This technique significantly enhances VSR usability for diverse, unencountered speakers.