British Sign Language Recognition via Late Fusion of Computer Vision and Leap Motion with Transfer Learning to...

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Sign language recognition is crucial for communication accessibility.
• Previous models often rely on single data modalities, limiting performance.
• Multimodality offers potential for richer feature extraction.

Purpose of the Study:

• To evaluate a late fusion multimodality approach for sign language recognition.
• To compare the performance of single-modality models against a fused model.
• To assess the effectiveness of transfer learning between sign languages.

Main Methods:

• Developed and benchmarked two deep neural networks: one for image classification (CNN and ANN) and one for Leap Motion data (evolutionary ANN search).
• Implemented a late fusion strategy to combine the outputs of the two best-performing single-modality networks.
• Utilized a large, synchronous dataset of 18 British Sign Language (BSL) gestures from multiple subjects.

Main Results:

• The fused multimodality model achieved 94.44% accuracy, outperforming individual image (88.14%) and Leap Motion (72.73%) models.
• The multimodality approach demonstrated superior performance on unseen data.
• Transfer learning with BSL weights improved American Sign Language (ASL) classification, with the fused model achieving 82.55% accuracy.

Conclusions:

• Late fusion of complementary features from different modalities significantly enhances sign language recognition.
• Multimodality is more robust than single-sensor methods, especially for unseen data.
• Transfer learning effectively improves cross-lingual sign language recognition performance.