3-D Markerless Tracking of Speech Movements With Submillimeter Accuracy

Area of Science:

• Speech Science
• Biomedical Engineering
• Machine Learning

Background:

• Accurate measurement of speech movements is crucial but challenging.
• Current methods often require physical sensors, which can be uncomfortable for children.
• Markerless facial tracking shows promise but lacks validated 3D precision.

Purpose of the Study:

• To develop and validate a novel markerless 3D lip tracking framework.
• To assess the precision and accuracy of the proposed method.
• To evaluate performance across different age groups.

Main Methods:

• Integration of a facial landmark detector with CoTracker, a transformer-based neural network.
• Joint tracking of dense facial points across video sequences.
• Validation against electromagnetic articulography data.

Main Results:

• The novel approach achieved high precision (≈ 0.15 mm SD), outperforming facial landmark detection alone (> 0.3 mm SD).
• 3D tracking accuracy was comparable to electromagnetic articulography (≈ 0.3 mm RMSE).
• Consistent performance was observed in adults and young children (3-4 years old).

Conclusions:

• The developed framework offers a precise and accurate markerless 3D lip tracking solution.
• This method is suitable for diverse populations, including young children.
• The approach promotes open science and has potential to enhance various markerless tracking applications in neuroscience.