Visual Speech Recognition with Lightweight Psychologically Motivated Gabor Features.

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Lip feature extraction is crucial for visual speech recognition.
• Current end-to-end methods offer good results but lack human interpretability.
• A need exists for explainable and efficient lip feature extraction techniques.

Purpose of the Study:

• To introduce a novel, lightweight, and human-centric lip feature extraction method.
• To demonstrate the effectiveness of these features for visual speech recognition using Long Short-Term Memory (LSTM) networks.
• To provide easily visualizable and analyzable features for better result comprehension.

Main Methods:

• Utilized Gabor-based image patches to extract 3D geometric lip features.
• Employed a human-centric approach inspired by psychological research on facial barcodes.
• Implemented LSTM-based machine learning for speech recognition tasks.

Main Results:

• Successfully extracted low-dimensionality lip parameters with minimal processing.
• Achieved a recognition rate of over 82% for overlapping speakers using the Grid corpus.
• Demonstrated comparable or superior performance to less explainable features in the literature.

Conclusions:

• The proposed Gabor-based feature extraction method is lightweight, efficient, and human-centric.
• These features enable reliable visual speech recognition with LSTM networks.
• The human-centric nature of the features enhances interpretability and analysis of results.