Biological batch normalisation: How intrinsic plasticity improves learning in deep neural networks.

Area of Science:

• Computational Neuroscience
• Deep Learning

Background:

• The Infomax principle guides information processing in neural systems.
• Controlling neuronal gain and bias is crucial for regulating firing rates.

Purpose of the Study:

• Introduce a novel local intrinsic plasticity (IP) rule inspired by Infomax.
• Investigate the IP rule's biological plausibility and compare it to batch normalization.
• Evaluate the IP rule's impact on deep network learning and gradient dynamics.

Main Methods:

• Developed the IP rule, which modulates neuronal gain and bias.
• Compared IP rule performance against batch normalization in deep networks.
• Analyzed error gradients during the learning process.

Main Results:

• The IP rule enhances deep network learning and robustness to synaptic learning rate increases.
• IP rule application significantly increases error gradient magnitudes, suggesting a solution to the vanishing gradient problem.
• IP rule yields neuronal information potential comparable to Infomax and improves upon batch normalization's information potential.

Conclusions:

• The IP rule offers a biologically plausible mechanism for improving deep learning.
• IP rule effectively mitigates the vanishing gradient problem.
• The IP rule demonstrates potential as an alternative or complement to batch normalization for enhancing neural network performance.