Evaluating the Potential Gain of Auditory and Audiovisual Speech-Predictive Coding Using Deep Learning.

Area of Science:

• Computational Neuroscience
• Machine Learning
• Speech Processing

Background:

• The brain's predictive coding framework suggests neurons process prediction errors between expected and actual stimuli.
• Limited data exists on the accuracy of predictions in real-world sensory environments.
• Focus on auditory and audiovisual speech processing within this predictive framework.

Discussion:

• Developed artificial neural network models (deep feedforward and convolutional) to predict future audio from past audio or audiovisual inputs.
• Models exploit local phonetic regularities without higher linguistic level integration.
• Experiments used the LibriSpeech and NTCD-TIMIT databases.

Key Insights:

• Models accurately predict speech stimuli within a short temporal range (25-50 ms), explaining 50-75% of variance.
• Predictive accuracy rapidly declines after 250 ms.
• Incorporating visual lip movement information improves prediction accuracy by 5-10%, with visual gains persisting longer than auditory ones.

Outlook:

• These findings suggest significant potential for computational savings in sensory processing.
• Further research can explore optimizing predictive models for enhanced speech perception.
• Investigating the temporal dynamics of audiovisual integration offers insights into neural efficiency.