An Improved Sign Language Translation Model with Explainable Adaptations for Processing Long Sign Sentences

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Sign language translation (SLT) aims to bridge communication gaps but struggles with long sign sentences due to dependencies and resource needs.
• Current neural SLT models exhibit limitations in handling complex, lengthy sign sequences effectively.

Purpose of the Study:

• To propose explainable adaptations for neural SLT models to improve performance on long sign sentences.
• To enhance the efficiency and accuracy of sign language translation systems.

Main Methods:

• Introduced a frame stream density compression (FSDC) algorithm to reduce redundant frames in sign language videos.
• Replaced the traditional NMT encoder with an improved architecture featuring temporal convolution (T-Conv) and dynamic hierarchical bidirectional GRU (DH-BiGRU) units.

Main Results:

• The proposed model demonstrated superior performance compared to state-of-the-art baselines on the RWTH-PHOENIX-Weather 2014T dataset.
• Achieved significant improvements, including up to 1.5+ BLEU-4 score gains, indicating enhanced translation quality.

Conclusions:

• The developed adaptations effectively address the challenges of processing long sign sentences in SLT.
• The optimized tokenization and improved encoder architecture lead to more accurate and resource-efficient sign language translation.