A Manifold Learning Perspective on Representation Learning: Learning Decoder and Representations without an Encoder.

Area of Science:

• Machine Learning
• Representation Learning
• Deep Learning

Background:

• Autoencoders, comprising an encoder and decoder, are standard for mapping high-dimensional data to lower-dimensional representations.
• The decoder defines a manifold in the input space, a concept relevant to manifold learning.

Purpose of the Study:

• To investigate training the decoder independently for representation learning.
• To determine the sample complexity of training decoders versus encoders.
• To enhance the conceptual understanding of representation learning.

Main Methods:

• Inspired by manifold learning, the decoder was trained independently using gradient descent.
• Mathematical expressions for sample requirements of encoder and decoder specification were derived.
• Performance was evaluated on MNIST, CIFAR10, and simulated gene regulatory data.

Main Results:

• The decoder requires significantly fewer training samples than the encoder for proper specification.
• Independent decoder training demonstrated superior performance in learning low-dimensional representations, particularly with small datasets.
• Experiments on image and gene regulatory data confirmed improved generalization and meaningful representations using decoder-only training.

Conclusions:

• Training the decoder alone is a viable and sample-efficient strategy for representation learning.
• This approach enhances generalization and understanding, especially beneficial for small datasets.
• The findings contribute to a better conceptual grasp of representation learning and autoencoder training.