Two-Stream Mixed Convolutional Neural Network for American Sign Language Recognition.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Convolutional Neural Networks (CNNs) excel in image recognition but struggle with feature correlation in dynamic gestures.
• Traditional CNN models are insufficient for recognizing complex correlations between time-consecutive images, posing challenges for American Sign Language (ASL) recognition, especially for gestures like 'J' and 'Z'.

Purpose of the Study:

• To propose a novel Two-Stream Mixed (TSM) method integrated with CNNs to enhance feature expression correlation for dynamic ASL gestures.
• To improve the accuracy and robustness of sign language recognition systems by addressing the limitations of traditional CNNs in handling temporal image data.

Main Methods:

• A Two-Stream Mixed (TSM)-CNN system was developed, comprising preprocessing, a TSM block for feature extraction and fusion, and CNN classifiers.
• The system utilizes two consecutive images as input streams, applying resizing, transformation, and augmentation during preprocessing.
• Feature maps are fused using addition and concatenation within the TSM block before classification.

Main Results:

• The TSM-CNN models demonstrated superior performance compared to standard CNN models without the TSM block.
• The TSM-ResNet50 model achieved the highest accuracy of 97.57% on both MNIST and ASL datasets.
• The proposed method effectively handles dynamic gestures, improving recognition of challenging ASL signs.

Conclusions:

• The Two-Stream Mixed (TSM) method significantly enhances CNN performance for dynamic gesture recognition in ASL.
• The TSM-ResNet50 model offers a highly accurate and viable solution for ASL recognition systems.
• This approach has potential applications in RGB image sensing systems for the hearing-impaired community.