Multimodal explainable AI predicts upcoming speech behavior in adults who stutter

Area of Science:

• Cognitive Neuroscience
• Neuroscience
• Artificial Intelligence

Background:

• Understanding the relationship between dynamic brain activity and behavior is a key goal in cognitive neuroscience.
• Brain activity can be measured directly (e.g., EEG) or indirectly through bodily signals like facial muscle movements.
• Previous AI research indicates both EEG and facial muscles encode information related to various cognitive and emotional states.

Purpose of the Study:

• To investigate the relationship between dynamic brain activity (EEG) and facial muscle activity in adults who stutter (AWS).
• To predict future speech behavior (fluent vs. stuttered) using multimodal data during speech preparation.
• To explore brain-behavior dynamics in AWS, who naturally fluctuate between fluent and stuttered speech.

Main Methods:

• Utilized electroencephalography (EEG) and facial muscle activity recorded via video during speech preparation in AWS.
• Employed an explainable self-supervised multimodal architecture to analyze temporal dynamics of EEG and facial muscle data.
• Trained the model to predict whether upcoming speech would be fluent or stuttered on a trial-by-trial basis.

Main Results:

• The multimodal architecture achieved 80.8% accuracy in predicting fluent or stuttered speech (chance=50%).
• Identified specific EEG and facial muscle signals that differentiate fluent from stuttered speech trials.
• Observed systematic variations in these signals across early and late speech preparation periods.

Conclusions:

• Dynamic brain and facial muscle activity patterns can predict upcoming speech behavior in AWS.
• The self-supervised multimodal approach effectively captured predictive brain-body dynamics.
• This methodology holds potential for understanding neural mechanisms in various neurological/psychiatric disorders and developing brain-state estimation technologies.