Search research articles

ABOUT JoVE

Overview Leadership Blog JoVE Help Center

AUTHORS

Publishing Process Editorial Board Scope & Policies Peer Review FAQ Submit

LIBRARIANS

Testimonials Subscriptions Access Resources Library Advisory Board FAQ

RESEARCH

JoVE Journal Methods Collections JoVE Encyclopedia of Experiments Archive

EDUCATION

JoVE Core JoVE Business JoVE Science Education JoVE Lab Manual Faculty Resource Center Faculty Site

Terms & Conditions of Use

Search research articles

Related Experiment Video

Updated: Sep 13, 2025

Capturing Dynamic Finger Gesturing with High-resolution Surface Electromyography and Computer Vision

Capturing Dynamic Finger Gesturing with High-resolution Surface Electromyography and Computer Vision

Published on: March 28, 2025

Fusion of Multimodal Spatio-Temporal Features and 3D Deformable Convolution Based on Sign Language Recognition in

Qian Zhou¹, Hui Li¹, Weizhi Meng²

¹School of Computer Science, Nanjing University of Posts and Telecommunications, 9 Wenyuan Road, Nanjing 210023, China.

Sensors (Basel, Switzerland)

|July 30, 2025

Summary

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by

Same author

Y-DWMS: A Digital Watermark Management System Based on Smart Contracts.

Sensors (Basel, Switzerland)·2019

Same author

[Association between genetic polymorphism of tumor necrosis factor and chronic severe hepatitis B in patients].

Zhonghua yi xue za zhi·2007

Same author

In vivo translational inaccuracy in Escherichia coli: missense reporting using extremely low activity mutants of Vibrio harveyi luciferase.

Biochemistry·2007

Same author

[Construction of recombinant adenovirus vector expressing extracellular domain of TbetaR-II-RANTES fusion gene and its anti-tumor effects].

Zhonghua zhong liu za zhi [Chinese journal of oncology]·2007

Same author

[Characteristics, evolution and variation of M genes of human avian H5N1 strains in Guangdong].

Bing du xue bao = Chinese journal of virology·2007

Same author

Dynamic changes in microbial activity and community structure during biodegradation of petroleum compounds: a laboratory experiment.

Journal of environmental sciences (China)·2007

This study introduces a novel deep learning framework for accurate sign language recognition (SLR) using skeleton and image data. The multimodal approach achieves competitive results on public datasets, advancing gesture interpretation.

Area of Science:

Computer Vision
Artificial Intelligence
Human-Computer Interaction

Background:

Sign language recognition (SLR) is challenging due to the complex spatio-temporal dynamics of visual language.
Existing methods often struggle with precise and efficient interpretation of raw video data.

Purpose of the Study:

To develop a novel multimodal deep learning framework for accurate and efficient sign language recognition.
To effectively integrate information from skeleton data and raw RGB images for improved SLR performance.

Main Methods:

A Multi-Stream Spatio-Temporal Graph Convolutional Network (MSGCN) was proposed for skeleton feature extraction, incorporating decoupling graph convolution, self-emphasizing temporal convolution, and spatio-temporal joint attention.
A 3D ResNet model with deformable convolution (D-ResNet) was utilized to process raw RGB image sequences.

Keywords:

ResNet2+1D multimodal fusion sign language recognition spatio-temporal graph convolutional network

More Related Videos

Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications

Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications

Published on: December 15, 2023

Related Experiment Videos

Last Updated: Sep 13, 2025

Capturing Dynamic Finger Gesturing with High-resolution Surface Electromyography and Computer Vision

Capturing Dynamic Finger Gesturing with High-resolution Surface Electromyography and Computer Vision

Published on: March 28, 2025

Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications

Author Spotlight: Enhancement of Salient Object Detection for Smart Grid Applications

Published on: December 15, 2023

A Multi-Stream Fusion Module (MFM) with a gating mechanism was employed to merge features from both modalities.

Main Results:

The proposed multimodal framework achieved competitive performance on the AUTSL and WLASL public datasets.
The integration of skeleton and RGB modalities through the MSGCN and D-ResNet models, fused by MFM, demonstrated effectiveness in SLR.

Conclusions:

The novel multimodal deep learning approach offers a promising direction for advancing sign language recognition technology.
The MSGCN and D-ResNet models, combined with the MFM, provide a robust solution for capturing complex spatio-temporal information in sign language gestures.