Neural Eigenfunctions are Structured Representation Learners.

Area of Science:

• Machine Learning
• Representation Learning
• Deep Learning

Background:

• Traditional spectral methods for unsupervised representation learning often lack scalability and out-of-sample generalization.
• Parametric modeling of eigenfunctions offers a path to overcome these limitations.

Purpose of the Study:

• To develop a scalable and generalizable method for learning structured representations without label supervision.
• To introduce a novel parametric approach to spectral representation learning using neural networks.

Main Methods:

• Parametrically modeling principal eigenfunctions of an integral operator using neural networks.
• Developing generalized objective functions for learning neural eigenfunctions, extending the EigenGame to function space.
• Utilizing data augmentation to derive similarity metrics, leading to a self-supervised learning objective with symmetry-breaking properties.

Main Results:

• The proposed method, Neural Eigenmap, learns structured, adaptive-length deep representations with features ordered by importance.
• In image retrieval, Neural Eigenmap achieved similar performance with up to 16x shorter representation length compared to leading self-supervised methods.
• Strong results were reported on a large-scale node representation learning benchmark with over one million nodes.

Conclusions:

• Neural Eigenmap offers an efficient and effective approach to unsupervised structured representation learning.
• The method demonstrates significant advantages in scalability, generalization, and representation compactness.
• Neural Eigenmap shows promise for applications in image retrieval and large-scale graph analysis.