[Development and evaluation of a deep learning algorithm for German word recognition from lip movements].

Area of Science:

• Artificial Intelligence
• Computer Vision
• Speech Technology

Background:

• Lip reading offers visual cues but is error-prone.
• Artificial neural networks (ANNs) enhance word recognition in lip reading.
• No AI-based lip-reading algorithms were available for the German language.

Purpose of the Study:

• To develop and evaluate an ANN for German lip-reading.
• To compare different neural network architectures and video processing techniques.
• To establish the accuracy of AI-driven lip reading for the German language.

Main Methods:

• Trained and validated a neural network using 1806 video clips of German speakers.
• Utilized 38,391 video segments with 32 speakers, focusing on 18 visually distinct polysyllabic words.
• Compared 3D Convolutional Neural Network (CNN), Gated Recurrent Units (GRU), and a combined GRUConv model, analyzing different image sections and color spaces.

Main Results:

• The GRUConv model achieved maximum accuracies of 87% with known speakers and 63% with unknown speakers.
• Cropping videos to the lips significantly improved accuracy to 70% compared to showing the entire face (34%).
• Color space variations showed minimal impact on classification rates (69%-72%).

Conclusions:

• The developed neural network represents the first AI-based lip-reading system for German.
• The algorithm demonstrates high accuracy, comparable to existing English-language systems.
• The model shows generalizability and effectiveness with unknown speakers, with potential for expansion to more word classes.