SignNet II: A Transformer-Based Two-Way Sign Language Translation Model

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Human-Computer Interaction

Background:

• Automated sign language processing (ASLP) traditionally focuses on sign-to-text, benefiting hearing individuals.
• Bridging the communication gap requires robust two-way translation between sign language and text.
• Transformer networks offer advanced capabilities for complex sequence-to-sequence tasks in ASLP.

Purpose of the Study:

• To introduce SignNet II, an AI architecture for facilitating two-way sign language communication.
• To enhance text-to-sign translation through metric embedding learning and sign similarity.
• To evaluate the performance of jointly trained networks against singly-trained models.

Main Methods:

• Development of SignNet II, a dual learning architecture comprising sign-to-text and text-to-sign networks.
• Joint training of networks using a dual learning mechanism.
• Implementation of metric embedding learning leveraging sign similarity for text-to-sign enhancement.
• Analysis of input feature representations using multi-feature transformers, focusing on keypoint-based pose features.

Main Results:

• SignNet II demonstrates significant advancements in two-way sign language processing.
• Jointly trained networks outperformed singly-trained counterparts in translation accuracy.
• Keypoint-based pose features showed consistent performance across varying video quality.
• Noteworthy enhancements in BLEU-1 - BLEU-4 scores were achieved on the German Sign Language (GSL) benchmark dataset.

Conclusions:

• SignNet II represents a promising step towards equitable two-way communication for Deaf and Hard of Hearing communities.
• Dual learning and metric embedding significantly improve ASLP performance, particularly for text-to-sign translation.
• The architecture's robustness with keypoint-based features makes it adaptable to real-world conditions.