Forward layer-wise learning of convolutional neural networks through separation index maximizing

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Traditional deep learning algorithms often rely on backpropagation, which can be computationally intensive and may struggle with vanishing gradients.
• Layer-wise training offers a potential alternative for optimizing complex neural network architectures, but effective methods are still under development.

Purpose of the Study:

• To propose a novel forward layer-wise learning algorithm for Convolutional Neural Networks (CNNs) in classification tasks.
• To introduce the Separation Index (SI) as a supervised complexity measure for evaluating and training individual network layers.
• To demonstrate improved feature representation by reducing data uncertainties and disturbances through a forward, layer-by-layer approach.

Main Methods:

• A forward layer-wise learning algorithm is proposed, utilizing the Separation Index (SI) to assess and train each layer sequentially.
• The SI is approximated using a variant of local triplet loss, enabling gradient-based maximization to enhance feature representation.
• The algorithm operates in a forward pass, inspired by the Neural Gradient Representation by Activity Differences (NGRAD) hypothesis, without requiring explicit error signals from the final layer.

Main Results:

• The proposed algorithm achieved superior performance in terms of accuracy compared to state-of-the-art methods across various image classification datasets (CIFAR-10, CIFAR-100, Raabin-WBC, Fashion-MNIST) and architectures (VGG16, VGG19, AlexNet, LeNet).
• Evaluations on text classification tasks (DBPedia, AG's News) also showed significant improvements.
• The method demonstrated enhanced time complexity, indicating greater efficiency.

Conclusions:

• The forward layer-wise learning algorithm effectively optimizes CNNs for classification by improving feature space representation.
• The Separation Index (SI) serves as a viable complexity measure for layer-wise training, outperforming existing methods.
• This approach offers a computationally efficient and accurate alternative for training deep neural networks.