Overparameterized neural networks implement associative memory.

Area of Science:

• Computational neuroscience
• Machine learning
• Deep neural networks

Background:

• Memorization and data retrieval are key computational problems.
• Understanding these mechanisms is crucial for both machine learning and neuroscience.

Purpose of the Study:

• To identify computational mechanisms for data memorization and retrieval.
• To investigate the role of overparameterized deep neural networks in memory.

Main Methods:

• Empirical evidence using overparameterized autoencoders.
• Theoretical analysis of autoencoders and sequence encoders.
• Trained deep neural networks using standard optimization methods.

Main Results:

• Overparameterized autoencoders store training samples as attractors, enabling sample recovery through iterative mapping.
• The same mechanism facilitates efficient encoding of example sequences.
• Theoretical proof that autoencoders store single examples as attractors.
• Sequence encoding is a more efficient memory mechanism than autoencoding.

Conclusions:

• Standard overparameterized deep neural networks implement a memorization and retrieval mechanism for real-valued data.
• This mechanism relies on attractor dynamics within the network.
• Sequence encoding offers a more efficient approach to memory compared to autoencoding.