Facial Expression Annotation and Analytics for Dysarthria Severity Classification.

Area of Science:

• Neurology
• Speech-Language Pathology
• Computer Science

Background:

• Post-stroke dysarthria often involves central facial paralysis, affecting motor control and emotional expression.
• Current assessments primarily use acoustic data, neglecting crucial visual facial cues and their emotional correlations.
• This limitation hinders comprehensive disease assessment and understanding of speech disorders.

Purpose of the Study:

• To develop a multimodal severity classification framework integrating facial and acoustic features for post-stroke dysarthria.
• To overcome data scarcity issues using a multi-level annotation algorithm.
• To enhance the objective assessment and clinical application of speech disorder evaluations.

Main Methods:

• A multi-level annotation algorithm utilizing a pre-trained model and motion amplitude was developed.
• Facial topology was modeled using Delaunay triangulation and spatial relationships captured via graph convolutional networks (GCNs).
• Abnormal muscle coordination was quantified using facial action units (AUs), and a multimodal feature fusion framework was proposed.

Main Results:

• The proposed multimodal framework achieved 92.0% accuracy and a 91.6% F1 score on the THE-POSSD dataset.
• Performance significantly surpassed single-modality baseline methods.
• The study identified changes in facial movements and sensitive areas related to emotional states in patients.

Conclusions:

• The multimodal framework effectively compensates for acoustic modality limitations with visual facial features.
• It demonstrates significant potential for objective assessment of speech disorders.
• This approach offers promising avenues for future clinical applications in evaluating post-stroke patients.