Related Experiment Video
Updated: Dec 16, 2025
05:56
Objectification of Tongue Diagnosis in Traditional Medicine, Data Analysis, and Study Application
Published on: April 14, 2023
3.0K
Predicting ultrasound tongue image from lip images using sequence to sequence learning
Kele Xu1, Jianqiao Zhao2, Boqing Zhu1
1National Key Laboratory of Parallel and Distributed Processing, National University of Defense Technology, Changsha, China.
The Journal of the Acoustical Society of America
|July 3, 2020
Summary
This study links lip shapes to tongue movements during speech using deep learning. The model accurately predicts tongue motion from lip videos, advancing speech science and understanding of speech production dynamics.
Area of Science:
- Speech Science
- Biomedical Engineering
- Machine Learning
Background:
- Understanding speech production dynamics is crucial for speech science.
- Simultaneous sensory streams, like lip shape and tongue movement, offer insights.
- The correlation between tongue deformation and lip shape is a key area of interest.
Purpose of the Study:
- To explore the association between lip shapes and tongue functional deformation during speech.
- To develop a predictive model linking visual lip movements to internal tongue dynamics.
Main Methods:
- Formulating the problem as a sequence-to-sequence learning task.
- Training a deep neural network using unlabeled lip videos.
- Predicting ultrasound tongue image sequences from lip video data.
Main Results:
- The machine learning model demonstrated satisfactory performance in predicting tongue motion.
- The study successfully established a learned association between lip imaging and ultrasound tongue imaging modalities.
Conclusions:
- Deep neural networks can effectively model the relationship between articulatory gestures (lip shape) and internal tongue movements.
- This approach advances the non-invasive study of speech production dynamics.
