Hebbian Descent: A Unified View on Log-Likelihood Learning | JoVE Visualize

Area of Science:

Artificial Intelligence
Machine Learning
Computational Neuroscience

Background:

The derivative of activation functions in output layers of artificial neural networks can negatively impact performance, especially in continual learning scenarios.
Vanishing error signals during backpropagation in saturated activation regions pose a challenge for training neural networks.

Purpose of the Study:

To propose Hebbian descent as a theoretical framework and practical implementation to address limitations caused by output layer activation function derivatives.
To introduce an alternative loss function, Hebbian descent loss, for gradient descent that bypasses the need for activation function derivatives.

Main Methods:

Developed Hebbian descent, an alternative weight update rule for the output layer that disregards the activation function's derivative.
Implemented Hebbian descent loss, equivalent to generalized log-likelihood loss.
Evaluated Hebbian descent in shallow and deep neural networks, particularly in continual learning tasks, with and without centering.

Main Results:

Hebbian descent effectively avoids vanishing error signals in saturated activation regions.
In continual learning for shallow networks, Hebbian descent outperformed Hebbian learning and other update rules, showing performance comparable to gradient descent.
With centering, Hebbian descent demonstrated superior prevention of catastrophic interference compared to other methods.
For deep neural networks, Hebbian descent exhibited comparable or better performance than standard gradient descent.

Conclusions:

Hebbian descent offers a unifying perspective on Hebbian learning, gradient descent, and generalized linear models.
The proposed method enhances continual learning capabilities and provides a robust alternative for training neural networks, especially those with saturating activation functions in the output layer.
Hebbian descent facilitates the design of effective loss-activation function combinations for improved neural network training.