Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Weakly supervised learning in video analysis presents challenges for sequence learning problems.
• Sign language recognition requires handling complex, parallel sub-problems like hand and mouth movements.

Purpose of the Study:

• To develop a new weakly supervised learning approach for video sequence analysis.
• To improve sign language recognition by exploiting parallel sub-problems and synchronization.

Main Methods:

• A novel approach using multi-stream Hidden Markov Models (HMMs) with intermediate synchronization constraints.
• Embedding Convolutional Neural Network-Long Short-Term Memory (CNN-LSTM) models within each HMM stream.
• Applying the method to sign language recognition, lip-reading, and hand shape recognition.

Main Results:

• The proposed method significantly outperforms the state-of-the-art on three benchmark datasets.
• Achieved superior performance in sign language recognition (over 1,000 classes), lip-reading, and hand shape recognition (over 60 shapes).
• Demonstrated significantly faster convergence rates due to the parallel alignment approach.

Conclusions:

• The novel multi-stream HMM approach with synchronization effectively handles parallel sequence learning in videos.
• This method offers a powerful and efficient solution for complex tasks like sign language recognition.
• The approach enables the discovery of subtle attributes for improved classification accuracy.