Stacking-Based Deep Neural Network: Deep Analytic Network for Pattern Classification.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks (DNNs) are typically trained end-to-end using backpropagation (BP), requiring significant computational resources like GPUs and large datasets.
• Stacking-based deep neural networks (S-DNNs) offer an alternative by aggregating independent learning modules, but their performance can be further optimized.
• Existing S-DNNs often lack efficient training methods for diverse datasets and may not fully leverage feature relearning capabilities.

Purpose of the Study:

• To introduce a novel ridge regression-based S-DNN, termed deep analytic network (DAN), and its kernelized version (K-DAN).
• To demonstrate DAN/K-DAN's capability for multilayer feature relearning from baseline and structured features.
• To evaluate the performance of DAN/K-DAN in pattern classification tasks across various domains.

Main Methods:

• Development of deep analytic network (DAN) using ridge regression for S-DNN layer training.
• Kernelization of DAN (K-DAN) to enhance feature relearning capabilities.
• Independent and decisive training of each S-DNN layer without backpropagation intervention.

Main Results:

• DAN/K-DAN effectively relearn features by perturbing intra/inter-class variations, alongside error reduction.
• Demonstrated superior performance of DAN/K-DAN on diverse pattern classification datasets (faces, handwritten digits, objects).
• DAN/K-DAN exhibit efficient trainability on CPUs, even with small datasets, unlike GPU-dependent BP-trained DNNs.

Conclusions:

• DAN/K-DAN represent a significant advancement in S-DNNs for pattern classification.
• These networks offer a more efficient and accessible alternative to traditional DNNs, particularly for resource-constrained environments.
• The proposed method outperforms existing S-DNNs and BP-trained DNNs without requiring data augmentation.