Daydreaming Hopfield Networks and their surprising effectiveness on correlated data

Area of Science:

• Artificial Intelligence
• Computational Neuroscience
• Machine Learning

Background:

• Hopfield networks are foundational recurrent neural networks for associative memory.
• Traditional Hopfield models face limitations in storage capacity and spurious memory formation.
• Existing dreaming algorithms offer partial solutions by erasing spurious memories.

Purpose of the Study:

• To develop an enhanced algorithm, Daydreaming, for improving the storage capacity and retrieval performance of Hopfield networks.
• To create a non-destructive learning algorithm that combines pattern reinforcement with spurious memory erasure.
• To investigate the algorithm's effectiveness on both uncorrelated and correlated data, including real-world datasets.

Main Methods:

• Developed a novel "Daydreaming" algorithm, integrating Hebbian-like reinforcement with spurious memory suppression.
• Trained the Daydreaming algorithm on random uncorrelated and correlated datasets generated via the random-features model.
• Evaluated the algorithm's performance on the MNIST handwritten digit dataset.

Main Results:

• Daydreaming demonstrates optimal performance with large basins of attraction and high-quality reconstruction for uncorrelated data.
• The algorithm effectively exploits data correlations, further increasing storage capacity and attractor stability.
• Daydreaming successfully stabilizes hidden data features and produces accurate attractors for unseen examples on MNIST.

Conclusions:

• The Daydreaming algorithm significantly enhances Hopfield network capabilities, offering superior storage capacity and memory retrieval.
• It effectively handles complex, correlated data and stabilizes underlying features, outperforming previous methods.
• Daydreaming shows practical applicability and robust performance on challenging real-world datasets like MNIST.