Glove-TalkII--a neural-network interface which maps gestures to parallel formant speech synthesizer controls

Area of Science:

• Human-Computer Interaction
• Speech Synthesis
• Assistive Technology

Background:

• Developing effective communication systems for individuals with speech impairments is crucial.
• Existing text-to-speech synthesizers often lack natural prosody and a direct user interface.
• Gesture-to-speech systems offer a potential alternative for augmenting or replacing vocal communication.

Purpose of the Study:

• To present Glove-TalkII, a novel system for translating hand gestures into synthesized speech.
• To demonstrate the system's capability for real-time speech production with an unlimited vocabulary.
• To evaluate the intelligibility and naturalness of speech produced by the Glove-TalkII system.

Main Methods:

• Glove-TalkII utilizes an adaptive interface to map continuous hand gestures to ten control parameters of a parallel formant speech synthesizer.
• The system employs three neural networks, including a gating network, for vowel and consonant production.
• Input devices such as Cyberglove, ContactGlove, a three-space tracker, and a foot pedal are integrated.

Main Results:

• One subject achieved intelligible speech output using Glove-TalkII.
• The system allows for direct control of fundamental frequency and volume, enhancing speech naturalness.
• Speech produced was slower than typical but exhibited more natural pitch variations compared to text-to-speech.

Conclusions:

• Glove-TalkII demonstrates a viable method for real-time speech synthesis directly from hand gestures.
• The adaptive interface and neural network architecture enable a flexible and potentially unlimited vocabulary.
• Further development could improve speech rate and expand user capabilities in assistive communication.