Empowering Deaf-Hearing Communication: Exploring Synergies between Predictive and Generative AI-Based Strategies...

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Effective communication between Deaf and hearing individuals is crucial for inclusivity.
• Existing digital solutions for sign language recognition (SLR) face challenges in cross-platform compatibility and continuous conversation support.
• Portuguese Sign Language (LGP) interpretation requires advanced methods for accurate and fluid communication.

Purpose of the Study:

• To propose a non-invasive LGP interpretation system-as-a-service.
• To enhance machine learning model training through dataset augmentation.
• To improve sign language tokenization and sentence construction for natural communication.

Main Methods:

• Leveraging Long-Short Term Memory (LSTM) architectures for skeletal posture sequence inference.
• Implementing dataset augmentation strategies to address data scarcity in machine learning.
• Developing a buffer-based interaction technique for real-time LGP term tokenization.
• Integrating a large language model (LLM) for human-like interpretation conditioning.

Main Results:

• LSTM models trained with 50 LGP terms and data augmentation achieved 80% to 95.6% accuracy.
• Users reported high intuitiveness with the buffer-based interaction strategy for tokenization.
• LLM integration (ChatGPT) showed promising semantic correlation rates in generated sentences.

Conclusions:

• The proposed system effectively interprets Portuguese Sign Language with high accuracy.
• The buffer-based interaction technique significantly improves the user experience and sentence coherence.
• The integration of LLMs holds potential for more natural and context-aware sign language interpretation.