OPTIMIZE WAV2VEC2S ARCHITECTURE FOR SMALL TRAINING SET THROUGH ANALYZING ITS PRE-TRAINED MODELS ATTENTION PATTERN

Area of Science:

• Speech Technology
• Machine Learning
• Medical Informatics

Background:

• Transformer-based automatic speech recognition (ASR) systems excel with large datasets.
• Medical research requires ASR for non-typical populations, such as children with speech disorders, often facing limited training data.
• Optimizing existing architectures like Wav2Vec 2.0 is crucial for efficient training on small datasets.

Purpose of the Study:

• To optimize the Wav2Vec 2.0 architecture for improved training efficiency on small datasets.
• To leverage block-level attention patterns in pre-trained models as indicators for architectural optimization.
• To enhance ASR performance for pre-school children with speech disorders under data-limited conditions.

Main Methods:

• Analyzed block-level attention patterns of pre-trained Wav2Vec 2.0 models.
• Optimized the Wav2Vec 2.0 architecture using local attention mechanisms and cross-block parameter sharing.
• Simulated limited data conditions using the Librispeech-100-clean dataset for reproducible experiments.

Main Results:

• Block-level attention patterns effectively guided the optimization process.
• The optimized Wav2Vec 2.0 architecture demonstrated improved performance compared to the vanilla version.
• Achieved an absolute word error rate (WER) reduction of 1.8% on dev-clean and 1.4% on test-clean datasets.

Conclusions:

• Block-level attention analysis is a viable strategy for optimizing Transformer-based ASR architectures on small datasets.
• The proposed optimization techniques enhance ASR performance for specialized populations with limited data.
• This research contributes to developing more effective ASR systems for children with speech disorders.