The SmartGlove BISINDO Wearable Sensor and Large Language Model-Based Sign Language-to-Speech System for Users With...

In Indonesia, people with communication disorders predominantly communicate using Bahasa Isyarat Indonesia (BISINDO). Nevertheless, all current systems for BISINDO recognition generate fragmented, word-by-word output and lack natural language generation (NLG) capabilities, thereby fundamentally constraining their communicative utility.

The aim of this study is to develop, validate, and field-evaluate SmartGlove BISINDO, an AI-assisted wearable sign language-to-speech system that embeds large language model (LLM)-powered NLG into a BISINDO translation pipeline for the first time.

This study followed the design science research (DSR) framework, covering phases 1 through 5 (problem identification, solution objectives, prototype design and development, demonstration, and evaluation). Data were collected from 12 purposively selected informants (6 domain experts and 6 deaf education companions/teachers) across 3 Indonesian cities (Malang, Surabaya, and Makassar), using a needs-analysis questionnaire and an expert validation sheet, and analyzed through statistical triangulation of agreement percentage, content validity ratio (CVR), and Aiken V. The validated prototype was field-tested in Makassar and Malang with 15 participants with communication disorders and 4 observers or companions; it was evaluated by a disability expert in Surabaya.

The needs analysis yielded a grand mean of 4.77 (SD 0.51) across 4 clusters, with 89% mean agreement, indicating very high priority for all identified needs. The prototype's 5-layer architecture integrates wearable sensors, on-device gesture classification, and LLM-based NLG. Expert validation showed an overall Aiken V of 0.94, with all 14 items exceeding the 0.75 validity cutoff and with 11/14 items (79%) rated highly valid (V>0.90). The grand mean CVR was 0.76 (SD 0.16), exceeding the Lawshe minimum threshold of 0.56 (P<.05), with 3 items-gesture differentiation, AI contextual relevance, and settings configuration ease-falling below the threshold and indicating areas for refinement. During the testing phase, the system received a lower practicality score (mean 3.35, SD 0.16; 67%), particularly for clarity of speech output. However, observers and companions gave a significantly higher score of 88%, with a mean of 4.40 (SD 0.08). A disability expert also gave a score of 81% (with a mean value calculated across the 20 questionnaire items of 4.05, SD 0.60). Regarding language, the AI-generated output underwent three transformation processes: addition, deletion, and substitution. Of the 3, substitution is the most likely to alter the intended meaning.

SmartGlove BISINDO is a wearable sign language-to-speech system that introduces the BISINDO research landscape to LLM-based NLG connect-and-translate approaches and demonstrates strong content validity. This study presents preliminary evidence toward a replicable model for interdisciplinary DSR-based prototype development that combines augmentative and alternative communication (AAC) theory, systemic functional linguistics, and design science methodology, intended as a reference for other underrepresented sign language contexts in Southeast Asia and beyond. Field evaluation further indicated favorable practicality overall, alongside a gap between expert-perceived feasibility and user-rated experience that will guide subsequent design refinement.