Encoding of speech in convolutional layers and the brain stem based on language experience

Area of Science:

• Neuroscience
• Artificial Intelligence
• Computational Linguistics

Background:

• Recent advances in computer vision and language models facilitate comparing artificial neural networks (ANNs) with neuroimaging data.
• Understanding spoken language representation in biological and artificial systems is crucial for advancing AI and neuroscience.

Purpose of the Study:

• To propose a novel framework for comparing biological and artificial neural computations of spoken language.
• To introduce new challenges and methodologies for comparing neural processing paradigms.

Main Methods:

• A technique based on electroencephalography (EEG) principles, averaging neural activity in the time domain.
• Direct comparison of brain responses (e.g., auditory brainstem response - ABR) and deep convolutional neural network (CNN) intermediate layer outputs to acoustic stimuli.
• Analysis of peak latency and the effect of language exposure on encoding.

Main Results:

• Biological and artificial neural responses to phonetic properties show high similarity without linear transformations.
• Quantified similarities between brainstem responses (cABR) and CNNs in encoding acoustic properties.
• Identified comparable peak latency encoding between human brain and CNNs across eight trained networks.

Conclusions:

• The proposed framework enables direct, transformation-free comparison of neural encoding between biological brains and ANNs.
• Significant similarities exist in how the human brain and CNNs process and encode spoken language features.
• This technique offers a versatile tool for comparing neural computations across different acoustic properties and neuroimaging modalities.