A multistream attention based neural network for visual speech recognition and sign language understanding.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Lip-reading recognition is crucial for visual speech understanding.
• Existing methods often struggle with the complexity of human movement.
• Sign language translation requires robust multi-modal analysis.

Purpose of the Study:

• Introduce SignKeyNet, a multi-stream keypoint-based neural network.
• Enhance lip-reading recognition and support sign language understanding.
• Model cross-modal spatiotemporal dependencies for improved accuracy.

Main Methods:

• Utilize pose estimation to extract 133 keypoints for hands, face, and body.
• Process each stream independently with specialized attention modules.
• Employ an attention-based fusion mechanism for multi-modal integration.

Main Results:

• Achieved 0.85 accuracy, 0.12 Word Error Rate (WER), and 0.06 Character Error Rate (CER) on the MIRACL-VC1 dataset.
• SignKeyNet outperformed baseline models including HMMs, DTW, CNNs, LSTMs, and Two-stream ConvNets.
• Demonstrated the effectiveness of attention-driven, multi-modal architectures.

Conclusions:

• SignKeyNet offers superior performance in lip-reading recognition.
• The architecture is extendable to full Sign Language Translation (SLT) systems.
• Shows potential for real-time accessibility technologies for the deaf and hard-of-hearing.