SignFormer-GCN: Continuous sign language translation using spatio-temporal graph convolutional networks.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Sign language is a vital communication system for the deaf and hard-of-hearing community.
• Existing Sign Language Translation (SLT) methods primarily rely on RGB features, which are limited by background and signer variations.
• Current SLT research often neglects the inherent graph structure of sign language, failing to capture low-level details.

Purpose of the Study:

• To enhance the accuracy and robustness of Sign Language Translation (SLT) systems.
• To address the limitations of using only RGB features in SLT.
• To incorporate the spatial-temporal dependencies of sign language into translation models.

Main Methods:

• Utilized a combination of keypoint and RGB features to better capture body part configurations.
• Employed a joint encoding technique integrating transformer and Spatio-Temporal Graph Convolutional Network (STGCN) architectures.
• Developed SignFormer-GCN to process both high-level context and low-level spatial-temporal graph structures.

Main Results:

• SignFormer-GCN achieved competitive performance on benchmark datasets: RWTH-PHOENIX-2014T, How2Sign, and BornilDB v1.0.
• The proposed method demonstrated effectiveness in enhancing translation accuracy across different sign language datasets.
• Experimental results validate the benefits of combining transformer and graph-based approaches for SLT.

Conclusions:

• The SignFormer-GCN model offers a significant advancement in Sign Language Translation by effectively capturing complex visual and structural information.
• Integrating spatial-temporal graph information alongside transformer-based context processing improves translation accuracy.
• This research provides a more comprehensive approach to SLT, paving the way for improved communication tools for the deaf community.