Convolutional and recurrent neural network for human activity recognition: Application on American sign language

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Human activity recognition faces challenges due to inter-subject and intra-subject variability.
• Robust time-series signal classification and validation are crucial for accurate activity recognition.

Purpose of the Study:

• To classify 60 American Sign Language (ASL) signs using LeapMotion sensor data.
• To evaluate conventional and deep learning models for sign language recognition.
• To introduce a DeepConvLSTM model integrating convolutional and recurrent layers.

Main Methods:

• Utilized LeapMotion sensor data for capturing hand and finger movements.
• Developed a kinematic model for the forearm, hand, fingers, and thumb.
• Applied data augmentation techniques to enhance neural network generalization.
• Compared DeepConvLSTM, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Multi-Layer Perceptrons (MLPs).

Main Results:

• DeepConvLSTM achieved 91.1 (3.8)% accuracy, outperforming other models.
• CNNs demonstrated high accuracy at 89.3 (4.0)%.
• The integration of convolutional layers proved effective for sign language recognition with depth sensor data.

Conclusions:

• Deep learning models, particularly DeepConvLSTM, show significant promise for accurate sign language recognition.
• Convolutional layers are beneficial for processing depth sensor data in sign language recognition tasks.
• The proposed methods offer a robust approach for time-series signal classification in human activity recognition.