Two-layer contractive encodings for learning stable nonlinear features

Area of Science:

• Machine Learning
• Deep Learning
• Artificial Intelligence

Background:

• Unsupervised learning of feature hierarchies is crucial for initializing deep learning architectures.
• Current methods often use greedy layer-by-layer approaches with auto-encoders or restricted Boltzmann machines.
• These methods employ linear projections and smooth thresholding, which can fail for certain computational tasks.

Purpose of the Study:

• To address the limitations of existing encoders in learning stable features for complex computations.
• To propose a more flexible two-layer encoder capable of learning a wider range of features.
• To improve the optimization and learning process for these enhanced encoders.

Main Methods:

• Developed a two-layer contractive encoder architecture.
• Extended contractive regularization techniques for the proposed encoder.
• Introduced linear transformations of hidden neurons to facilitate learning.
• Evaluated performance on artificial and benchmark datasets, including a semi-supervised task.

Main Results:

• Demonstrated that existing layer-by-layer encoders struggle with exclusive-or computations.
• The proposed two-layer contractive encoder successfully learns stable features for challenging datasets.
• Linear transformation of hidden neurons improved the learning process for the encoder.
• Positive results were observed in both unsupervised and semi-supervised learning tasks.

Conclusions:

• The proposed two-layer contractive encoder offers a more robust approach to unsupervised feature hierarchy learning.
• This method overcomes limitations of traditional deep learning initialization techniques.
• The approach shows significant promise for improving deep learning performance, particularly in semi-supervised settings.