Intermediate acoustic-to-semantic representations link behavioral and neural responses to natural sounds

Area of Science:

• Neuroscience
• Computational Auditory Neuroscience
• Machine Learning in Neuroscience

Background:

• Sound recognition involves transforming auditory waveforms into semantic representations.
• The superior temporal gyrus (STG) is a key brain region, but its computational role in sound processing is unclear.
• Understanding these transformations is vital for deciphering auditory perception.

Purpose of the Study:

• To characterize the computational mechanisms underlying sound recognition in the human auditory cortex.
• To compare acoustic, semantic, and deep neural network models in predicting brain activity and perceived sound dissimilarity.
• To elucidate the nature of sound representations within the STG.

Main Methods:

• Utilized a model comparison framework to evaluate different computational models.
• Collected 7 Tesla functional magnetic resonance imaging (fMRI) data from human participants listening to sounds.
• Assessed the predictive power of models for auditory cortex responses and subjective sound dissimilarity ratings.

Main Results:

• Spectrotemporal modulations predicted early auditory cortex (Heschl's gyrus) responses.
• Auditory dimensions like loudness predicted STG responses and perceived dissimilarity.
• Sound-to-event deep neural networks significantly outperformed acoustic and semantic models in predicting STG responses and perceived dissimilarity.

Conclusions:

• The STG processes intermediate acoustic-to-semantic sound representations not captured by simpler models.
• These representations are compositional and behaviorally relevant.
• Deep neural networks offer a powerful tool for modeling complex auditory processing in the brain.