Dynamic Perturbation of Weights for Improved Data Reconstruction in Unsupervised Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Data Science

Background:

• Weight pruning is established for neural network compression in supervised learning.
• Its application in unsupervised learning, particularly for autoencoders with non-imaging data, remains underexplored.

Purpose of the Study:

• To investigate the efficacy of weight pruning for unsupervised autoencoder models using non-imaging data.
• To propose a deterministic algorithm for model perturbation to improve data reconstruction and compression.

Main Methods:

• Adapting weight pruning for autoencoder weight dynamics during data reconstruction.
• Developing a deterministic model perturbation algorithm using weight statistics and binary masks.
• Experimenting across eight diverse non-imaging datasets (e.g., gene sequences, swarm behavior).

Main Results:

• Periodic weight perturbations improved autoencoder data reconstruction accuracy and model compression.
• A small fraction (<5%) of weights were found to be highly informative, outperforming negative weights (>90%).
• Perturbing low or negative weights generally reduced data reconstruction loss.

Conclusions:

• Weight pruning is a viable technique for enhancing unsupervised autoencoder performance on non-imaging data.
• The proposed perturbation method offers a deterministic approach to optimize autoencoder behavior for better data representation.
• This work contributes to understanding and correcting dynamic behaviors in neural networks for improved reconstruction and latent variable discovery.