Perturbation of deep autoencoder weights for model compression and classification of tabular data

Area of Science:

• Machine Learning
• Deep Learning
• Data Science

Background:

• Fully connected deep neural networks (DNNs) suffer from redundant weights, leading to overfitting and high memory usage.
• DNNs often underperform traditional machine learning models in tabular data classification tasks.

Purpose of the Study:

• To propose a novel weight perturbation strategy for DNNs, focusing on self-supervised pre-training of deep autoencoders.
• To improve DNN performance and reduce model size for tabular data classification.

Main Methods:

• Implemented periodic weight perturbations (prune and regrow) during the self-supervised pre-training of deep autoencoders.
• Compared the proposed method against dropout learning and weight regularization (L1/L2).
• Evaluated performance on six diverse tabular data sets for downstream classification tasks.

Main Results:

• The proposed weight perturbation strategy outperformed dropout and regularization on four out of six tabular data sets.
• Achieved 15% to 40% sparsity, compressing pretrained models without performance loss, unlike traditional pruning.
• Pretrained deep autoencoders with weight perturbation surpassed traditional machine learning models in several tabular data classification scenarios.

Conclusions:

• Periodic weight perturbation is an effective method for DNN compression and performance enhancement in tabular data classification.
• The effectiveness of deep models on tabular data depends on variable characteristics; traditional models excel with uncorrelated variables.
• This approach offers a competitive alternative to traditional machine learning for specific tabular data challenges.