Enhancing english oral translation through cross-modal learning and synchronous optimization

Area of Science:

• Natural Language Processing
• Computational Linguistics
• Artificial Intelligence

Background:

• Spoken English translation faces challenges due to pronunciation variations and rapid speech.
• Current synchronous translation methods lack efficacy in real-time oral contexts.

Purpose of the Study:

• To enhance the quality and efficiency of synchronous English oral translation.
• To explore the integration of cross-modal semantic understanding and synchronous enhancement.

Main Methods:

• Implemented a cross-modal translation scenario.
• Utilized Bidirectional Encoder Representations from Transformers (BERT) to amalgamate text sequences with speech features.
• Optimized Transformer's self-attention mechanism for context-awareness and dynamic time warping (DTW) for real-time synchronization.

Main Results:

• Achieved a 9.3% and 26.9% higher average Bilingual Evaluation Understudy (BLEU) score compared to existing models.
• Demonstrated a 17.9% and 16.8% faster synchronization speed than other models.
• The proposed fusion model significantly improved translation fluency and accuracy.

Conclusions:

• The fusion model, incorporating context-awareness and attention mechanisms in cross-modal translation, elevates English oral translation quality.
• This approach offers a novel solution for synchronous spoken English translation.
• The findings highlight the potential of advanced AI models in overcoming real-time translation barriers.