Explainable AI for sign language recognition models: Integrating Grad-Cam LIME and Integrated Gradients

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Sign language recognition is vital for communication between hearing and deaf communities.
• Existing models often struggle with noise and lack transparency.

Purpose of the Study:

• To enhance the robustness and explainability of a VGG19-based sign language classification model.
• To introduce a segmentation-based approach using hand masks and validate it with Explainable Artificial Intelligence (XAI).

Main Methods:

• Dataset augmentation and alternative data representations.
• A segmentation-based approach using U-Net generated hand masks, replacing depth images.
• Explainable Artificial Intelligence (XAI) methods including Grad-CAM, LIME, and integrated gradients for model interpretation.

Main Results:

• The mask-based model demonstrated improved classification accuracy compared to depth images by focusing on hand shape and structure.
• Comparative analysis showed RGB models capture texture/color, while mask-based models focus on essential hand features.
• XAI methods validated results, highlighting influential image regions and enabling multi-perspective analysis.

Conclusions:

• The enhanced model improves generalization and explainability for American Sign Language recognition.
• The mask-based approach with XAI integration increases transparency and reliability in assistive technologies.
• This research fosters greater trust and usability in sign language recognition systems.