Thrivaad: A Multilingual, Predictive Eye-Sign-Based AAC System Powered by Optimized Deep Learning

Around 1.5% of the global population is suffering from speech impairments; the major causes for this are cerebral palsy and ALS, and the only way for these individuals to communicate is through Augmentative and Alternative Communication (AAC). These systems are either electronic or non-electronic. Based on new study developments, electronic methods, such as Brain-Computer Interaction (BCI) and eye-gaze-based communication, are assessed as the best choices, but they have their own limitations, incorporating limited adaptability to changing conditions, such as setup variations and user fatigue, which reduces the system's robustness. Our previous study, Netravad, shows potential for addressing these gaps, but it lacks multilingual support and will not yield the same results under changing lighting conditions. This study, Thrivaad, provides multilingual support and text prediction and integrates optimized deep learning to accurately capture eye movements even in varying environmental lighting conditions. Thrivaad uses eye movements as input from a webcam, and the Optuna-optimized YOLOv5 model is used to detect the eye direction accurately. Then communication is established in English, Malayalam, and Hindi. The text-prediction feature of this system improves communication by reducing the number of eye gestures required to form a message. This study included a total of 60 participants across three age groups with 35,263 eye-sign images collected. With this data, the YOLOv5 model is trained and then optimized by Optuna. The proposal system provides accurate eye direction, text prediction, multilingual support, and improved adaptability to changing conditions for eye-based AAC.