AI-NLP powered assistive technology system for individuals with vocally and hearing impairments

This paper addresses communication challenges for vocally and hearing impaired individuals by developing a cost-effective, high-accuracy device utilizing deep learning and Natural Language Processing (NLP). The device supports interaction by recognizing both Indian Sign Language and Customized Sign Languages. We evaluated four AI models, including Random Forest Classifier, XGBoost Classifier, Support Vector Machine (SVM), and Convolutional Neural Network (CNN). The CNN achieved the highest accuracy of 90%, effectively capturing intricate sign language gestures, compared to 80% for SVM, 81% for Random Forest and 76% for XGBoost. Consequently, CNN was deployed in the device. The system features an embedded System on Chip board for affordability and operates in two phases: interpreting hand gestures via CNN and converting them into voice commands through NLP, delivered via speaker or earphones. A mobile app is included to enhance communication between impaired and non-impaired users. This solution aims to bridge communication gaps and improves the quality of life for the hearing and vocally impaired.