Expectation Learning for Stimulus Prediction Across Modalities Improves Unisensory Classification.

Area of Science:

• Computational neuroscience
• Machine learning
• Artificial intelligence

Background:

• Expectation learning enhances perception by associating multisensory inputs, like linking sounds to sights.
• Humans naturally refine these associations over time without explicit external supervision, relying on temporal co-occurrence.

Purpose of the Study:

• To develop a computational model for unsupervised expectation learning.
• To capture key properties of expectation learning, particularly the absence of explicit supervision beyond temporal co-occurrence.

Main Methods:

• A novel hybrid neural model combining audio-visual autoencoders and a recurrent self-organizing network was developed.
• The model facilitates stimulus prediction across modalities, enabling cross-sensory reconstructions.
• This mechanism is termed stimulus prediction across modalities.

Main Results:

• The proposed model demonstrated the capability to learn concept bindings effectively.
• Evaluation on unisensory classification tasks using audio-visual stimuli from the AudioSet corpus validated the model's performance.
• The model successfully learned associations from 43,500 YouTube videos.

Conclusions:

• The developed hybrid neural model successfully implements unsupervised expectation learning.
• The model's ability to predict stimuli across modalities showcases a promising approach for artificial sensory perception.
• This work contributes to understanding and modeling unsupervised learning in artificial systems.