Natural sounds can be reconstructed from human neuroimaging data using deep neural network representation.
Jong-Yun Park1,2, Mitsuaki Tsukamoto2, Misato Tanaka1,2
1Department of Intelligence Science and Technology, Graduate School of Informatics, Kyoto University, Kyoto, Japan.
Plos Biology
|July 23, 2025
Summary
Researchers developed a new method to reconstruct sounds from brain activity using deep neural networks (DNNs) and functional MRI (fMRI). This approach successfully decodes auditory features, enabling plausible sound reconstructions despite fMRI
Area of Science:
- Neuroscience
- Auditory Perception
- Machine Learning
Background:
- Decoding visual information from brain activity using fMRI is advanced, but auditory reconstruction remains difficult due to temporal complexities and fMRI limitations.
- Deep neural networks (DNNs) offer hierarchical auditory features that may bridge the gap in reconstructing complex soundscapes.
- Understanding neural representations of sound is crucial for advancing brain-computer interfaces and auditory prosthetics.
Purpose of the Study:
- To develop and evaluate a novel method for reconstructing auditory experiences from brain activity.
- To investigate the efficacy of using decoded deep neural network (DNN) features for sound reconstruction.
- To assess the perceptual plausibility and generalization capabilities of the reconstructed sounds.
Main Methods:
- Integrated brain decoding of DNN-derived auditory features with an audio-generative model.
- Utilized functional MRI (fMRI) data to capture population neural responses in the auditory cortex.
- Compared decoded DNN features against traditional spectrotemporal and modulation-based features.
Main Results:
- Decoded DNN features significantly outperformed other feature types, enabling perceptually plausible sound reconstructions across various categories.
- Reconstructions accurately captured short-term spectral properties and timbre (e.g., speech, music), but not long temporal sequences.
- The method demonstrated generalization across sound categories and reflected attended sounds during a selective auditory attention task.
Conclusions:
- The proposed framework successfully maps brain activity to auditory experiences, advancing sound reconstruction from neural data.
- DNN features are effective for decoding auditory information, offering a promising avenue for future auditory research and applications.
- While limited by fMRI's temporal resolution, the approach represents a significant step towards understanding and reconstructing internal auditory representations.
