Improving Network Training on Resource-Constrained Devices via Habituation Normalization.

Area of Science:

• Deep Learning
• Computational Neuroscience
• Artificial Intelligence

Background:

• Batch Normalization (BN) is crucial for deep learning but struggles with small batch sizes, leading to accuracy loss.
• Resource-constrained devices often require small batch training, hindering BN's effectiveness.
• The fruit fly olfactory system utilizes a "negative image" habituation model for information filtering and stability.

Purpose of the Study:

• To introduce a novel normalization method, Habituation Normalization (HN), inspired by biological neural circuits.
• To address the accuracy degradation of BN with small batch sizes in deep learning.
• To evaluate HN's performance against standard normalization techniques and its applicability in real-world scenarios.

Main Methods:

• Developed Habituation Normalization (HN) by first eliminating "negative images" from habituation, then computing normalization statistics.
• Implemented and tested HN on LeNet-5, VGG16, and ResNet-50 architectures using Fashion MNIST and CIFAR10 datasets.
• Compared HN's robustness and accuracy across various batch sizes against four established normalization methods.

Main Results:

• HN demonstrates stable and high accuracy across different batch sizes, outperforming standard methods.
• Experiments show HN accelerates neural network training and boosts model accuracy.
• HN proved effective in deep learning-based EEG signal processing, suitable for fine-tuning and applications with limited resources.

Conclusions:

• Habituation Normalization (HN) offers a robust and efficient alternative to traditional normalization methods, particularly for small batch sizes.
• HN's bio-inspired approach enhances deep learning model performance and stability.
• HN is well-suited for practical applications on devices with limited computational power and memory.