Dissecting neural computations in the human auditory pathway using deep neural networks for speech

Area of Science:

• Neuroscience
• Computational Auditory Neuroscience
• Artificial Intelligence

Background:

• Traditional linear models have limited success in explaining complex speech processing by the human auditory system.
• Artificial neural networks (ANNs) show promise for modeling speech recognition and processing.
• Understanding neural coding from the auditory nerve to the speech cortex is crucial for deciphering speech perception.

Purpose of the Study:

• To investigate neural coding in the human auditory system using speech representations from deep neural network (DNN) models.
• To explore the correlation between DNN layer representations and neural activity across the auditory pathway.
• To determine if unsupervised speech models can effectively model auditory processing.

Main Methods:

• Utilized state-of-the-art deep neural network (DNN) models with speech representations.
• Correlated hierarchical DNN layer representations with neural activity data from the auditory nerve to the speech cortex.
• Compared unsupervised DNN models with supervised and fine-tuned models for speech processing tasks.
• Investigated DNN model predictions of cortical responses in native English and Mandarin speakers.

Main Results:

• Hierarchical layers of DNN models showed strong correlations with neural activity throughout the ascending auditory system.
• Unsupervised speech models performed comparably to supervised or fine-tuned models.
• Deeper DNN layers exhibited better correlation with higher-order auditory cortex activity, aligning with phonemic and syllabic structures.
• DNN models trained on English or Mandarin successfully predicted cortical responses in native speakers of those languages.

Conclusions:

• DNN model representations converge with the biological auditory pathway, offering a powerful framework for understanding speech processing.
• This study provides new computational approaches for modeling neural coding in the auditory cortex.
• The findings highlight the potential of ANNs in elucidating complex neural processes in auditory perception.