End-to-end Jordanian dialect speech-to-text self-supervised learning framework.

Area of Science:

• Natural Language Processing
• Speech Recognition
• Artificial Intelligence

Background:

• Speech-to-text (STT) technology is crucial for human-robot interaction.
• Low-resource languages, including Arabic dialects, lack sufficient labeled speech data.
• Self-supervised and noisy student training offer promising solutions for data scarcity.

Purpose of the Study:

• To develop an end-to-end, transformers-based framework for low-resource STT systems.
• To create an efficient speech-to-text system for the Jordanian Arabic dialect.
• To leverage unlabeled data through self-supervised and noisy student training.

Main Methods:

• An end-to-end transformers-based model incorporating customized audio-to-text processing.
• Framework for ingesting data from multiple sources and accelerating manual annotation.
• Utilizing noisy student training and self-supervised learning with data augmentation for pre- and post-training.

Main Results:

• The proposed self-training approach achieved a 5% word error rate reduction compared to fine-tuned Wav2Vec.
• Successfully developed a Jordanian Arabic dialect speech-to-text system.
• Demonstrated the framework's ability to utilize unlabeled data with minimal human intervention.

Conclusions:

• The research provides a valuable Jordanian-spoken dataset and an effective end-to-end approach for low-resource STT.
• The framework enables the development of new Arabic STT applications, such as question-answering and intelligent control systems.
• This work enhances human-robot interaction by adding human-like perception and hearing capabilities to robots.