Vision transformer-powered conversational agent for real-time Indian Sign Language e-governance accessibility.

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• The Deaf and Hard-of-Hearing community faces significant digital accessibility barriers, particularly in accessing e-governance services.
• Indian Sign Language (ISL) is a low-resource language, posing challenges for developing effective digital tools.
• Existing systems often lack the context-awareness and real-time processing needed for natural conversational interactions.

Purpose of the Study:

• To develop a context-aware, real-time Indian Sign Language (ISL) conversational agent.
• To enhance digital accessibility for the Deaf and Hard-of-Hearing community in e-governance.
• To support continuous ISL recognition from isolated words to full sentences.

Main Methods:

• Utilized a hybrid architecture combining Convolutional Neural Networks (CNNs) with Vision Transformer (ViT) models.
• Developed a domain-specific ISL video dataset to address low-resource challenges.
• Implemented a dynamic context-response mapping engine for improved accuracy with ambiguous inputs.

Main Results:

• Achieved 97.5% accuracy under optimized conditions.
• Demonstrated a mean response time of 6.53 seconds.
• Attained an average System Usability Scale (SUS) score of 71.83.

Conclusions:

• The developed ISL conversational agent significantly enhances digital inclusion for the Deaf and Hard-of-Hearing community.
• The system's modular design and context-aware engine ensure scalability and reliability for real-world deployment.
• This advancement facilitates real-time, inclusive digital interactions within e-governance services.