Skeleton-based Chinese sign language recognition and generation for bidirectional communication between deaf and...

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision

Background:

• Chinese Sign Language (CSL) is a widely used sign language, yet current technologies primarily focus on recognition, limiting bidirectional communication.
• Existing sign language recognition (SLR) systems address only one-way communication, necessitating advancements in sign language generation (SLG) for a complete communication bridge.

Purpose of the Study:

• To propose a novel skeleton-based framework for bidirectional Chinese Sign Language (CSL) communication, integrating both recognition and generation.
• To develop a two-level probability generative model capable of producing diverse and recognizable CSL skeletal sequences, addressing data scarcity.

Main Methods:

• A recurrent neural network (RNN) based framework was developed for skeleton-based CSL recognition and generation.
• A two-level probability generative model with a flexible approximate posterior distribution was employed.
• Experiments were conducted on the large 500 CSL dataset to evaluate bidirectional communication capabilities.

Main Results:

• The proposed framework achieved high recognition accuracy for both real and synthetic CSL data.
• The generation method effectively compensated for limited training data, improving overall performance.
• A reduced runtime was observed, indicating computational efficiency.

Conclusions:

• The developed bidirectional communication framework and generation algorithm represent an effective new approach for CSL recognition and generation.
• The model's ability to generate stylized skeletal sequences enhances human recognizability and aids in overcoming data limitations.
• This framework has potential applications beyond CSL, extending to other sequence-to-sequence information interactions.