Facial Movements Extracted from Video for the Kinematic Classification of Speech

Area of Science:

• Speech-language pathology
• Biomedical engineering
• Computer vision

Background:

• Speech Sound Disorders (SSDs) significantly impact children's academic and social development.
• Accurate diagnosis of SSDs is crucial for effective intervention and mitigating long-term effects.
• Current diagnostic methods for motor speech control issues can be subjective and time-consuming.

Purpose of the Study:

• To evaluate the feasibility of using automated 3D facial measurements from 2D videos for classifying speech movements.
• To assess the potential of the Speech Movement and Acoustic Analysis Tracking (SMAAT) system for objective SSD assessment.
• To determine if inferred depth measurements from 2D video can aid in speech movement analysis.

Main Methods:

• Collected 2D front-facing videos of 51 adults and 77 typically developing children (3-4 years old) speaking 20 words.
• Utilized a facial mesh detection and tracking algorithm to extract 3D facial measurements (jaw, lips) from video frames.
• Employed Leave-One-Out Cross-Validation (LOOCV) to test the word classification performance of individual measurements.

Main Results:

• Several automatically extracted facial measurements demonstrated significant word classification performance in both adult and child cohorts.
• Inferred depth measurements from 2D video were found to be significant predictors of speech movements.
• Classified measurements aligned with expected facial movements, validating their clinical relevance.

Conclusions:

• Automated 3D facial measurements from standard 2D videos show promise for objective speech movement analysis.
• The SMAAT system has potential as a tool to support clinical diagnosis of motor speech control issues in SSD.
• This approach offers a feasible method for rapid and objective assessment of speech production.