Robust neural tracking of linguistic speech representations using a convolutional neural network.

Area of Science:

• Auditory neuroscience
• Computational linguistics
• Brain-computer interfaces

Background:

• Neural activity in the brain tracks various speech signal features during auditory perception.
• Electroencephalography (EEG) combined with speech signals can measure neural tracking.
• Linear models show linguistic features contribute to neural tracking but cannot capture nonlinear brain dynamics.

Purpose of the Study:

• To develop and evaluate a nonlinear convolutional neural network (CNN) model for analyzing neural tracking of linguistic features in speech.
• To investigate the contribution of specific linguistic features (phoneme surprisal, cohort entropy, word surprisal, word frequency) beyond lexical information.
• To compare the performance of the nonlinear CNN against linear encoder and linearized CNN models.

Main Methods:

• Integration of phoneme- and word-based linguistic features within a nonlinear CNN framework.
• Utilizing phoneme and word onsets as controls to isolate linguistic information.
• Comparative analysis of the nonlinear CNN against linear encoder and linearized CNN models.

Main Results:

• The nonlinear CNN demonstrated a significant contribution of cohort entropy (CE) over phoneme onsets.
• Word surprisal (WS) and word frequency (WF) showed significant contributions over word onsets.
• The nonlinear CNN model consistently outperformed the linear baseline models.

Conclusions:

• Nonlinear modeling, specifically with CNNs, offers a more sensitive approach to measuring the neural coding of linguistic information in speech.
• This nonlinear approach reveals effects that may be undetectable with linear models, potentially improving objective speech understanding measures.
• The findings suggest potential for enhanced within-subject speech perception analysis and reduced recording durations in future auditory neuroscience research.