Deep Sound Synthesis Matched to Brain Activity Recapitulates Preferential Responses to Speech and Music

Area of Science:

• Neuroscience
• Auditory Perception
• Computational Auditory Neuroscience

Background:

• The human auditory system categorizes environmental sounds into semantic groups like speech and music.
• Distinct acoustic features and auditory cortex regions are involved, but internal representations are poorly understood.

Purpose of the Study:

• Investigate internal sound features in the auditory cortex for speech and music categorization.
• Understand the functional role of these representations in sound categorization.

Main Methods:

• Combined neuroimaging, deep neural network (DNN) analysis, brain-based sound synthesis, and psychophysical testing.
• Synthesized sounds from cortical activity patterns to test auditory cortex responses.
• Examined human participants' behavioral and neural responses to synthetic sounds.

Main Results:

• Sounds synthesized from cortical activity elicited similar categorical responses as natural speech and music.
• These synthetic sounds were acoustically dissimilar to natural speech and music.
• Auditory cortex responses suggest reliance on abstracted, internal category structures.

Conclusions:

• Auditory categorization relies on internal, abstracted representations not reducible to natural acoustic properties.
• Deep neural networks (DNNs) capture intermediate sound features relevant to auditory categorization.
• Findings offer new insights into the neural basis of sound meaning extraction.