A data-efficient and easy-to-use lip language interface based on wearable motion capture and speech movement...

Area of Science:

• Human-Computer Interaction
• Speech Processing
• Wearable Technology

Background:

• Current lip reading technologies lack reliability, require extensive data, and generalize poorly.
• There is a need for user-friendly, wearable interfaces for high-fidelity lip movement acquisition.

Purpose of the Study:

• To develop a wearable lip language decoding technology for interference-free, high-fidelity lip reading.
• To create data-efficient decoder-modeling methods for fluent lip language recognition.

Main Methods:

• Utilized wearable motion capture for lip movement acquisition.
• Developed a method for continuous lip speech movement reconstruction.
• Artificially generated continuous speech datasets from limited user word samples.

Main Results:

• Achieved 92.0% average accuracy in continuous, fluent lip speech recognition across seven individuals.
• Successfully recognized 93 English sentences without user-specific training.
• Demonstrated no training burden on users due to artificially generated datasets.

Conclusions:

• The proposed method offers a data-efficient and user-friendly paradigm for lip language recognition.
• Wearable technology combined with data augmentation significantly improves lip reading accuracy and accessibility.
• Minimizes user training load, making advanced lip reading technology more practical.