A comparative study of nonlinear cleanup rules in resonator networks

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Hopfield networks are known for pattern retrieval via nonlinear updates.
• Associative memory variants expanded on Hopfield networks with novel retrieval rules.
• Resonator networks offer an analogous iterative approach for vector-symbolic factorization.

Purpose of the Study:

• To investigate the impact of different cleanup nonlinearities on resonator network performance.
• To analyze the factorization capacity, failure modes, and convergence of various resonator network variants.
• To establish resonator networks as a family of Hopfield-inspired factorization dynamics.

Main Methods:

• Studied four resonator network variants: sign, ReLU, polynomial, and softmax cleanup.
• Evaluated factorization capacity, terminal failure modes, convergence, and empirical complexity.
• Assessed hyperparameter sensitivity, FHRR behavior, sign-projection choices, and update noise.
• Analyzed neural-output decomposition on visual scene data.

Main Results:

• The choice of cleanup nonlinearity critically affects the capacity transition and dominant failure modes.
• Different nonlinearities lead to distinct convergence behaviors and empirical complexities.
• Hyperparameter sensitivity and internal update noise vary across the studied variants.
• Resonator network performance is sensitive to sign-projection choices and dimensionality.

Conclusions:

• Resonator networks should be considered a family of factorization dynamics, not a single rule.
• The cleanup nonlinearity is a key factor in tailoring resonator networks for specific tasks.
• These findings extend the understanding of Hopfield-inspired models for complex factorization problems.