Sparsely Activated Networks.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Data Science

Background:

• Unsupervised learning literature often overlooks representation description length, a key indicator of model complexity.
• Existing methods prioritize feature learning without a direct measure of representational efficiency.

Purpose of the Study:

• Introduce a novel metric (φ) to evaluate unsupervised models based on reconstruction accuracy and representation compression.
• Develop and assess sparsely activated networks (SANs) that minimize the φ metric.
• Investigate the interpretability and efficiency of learned representations.

Main Methods:

• Defined the φ metric incorporating reconstruction accuracy and description length.
• Evaluated standard (Identity, ReLU) and novel sparse activation functions (top-k absolutes, Extrema-Pool indices, Extrema).
• Proposed and implemented sparsely activated networks (SANs) with shared weights and sparse activation.

Main Results:

• Models selected using the φ metric demonstrated minimal description length.
• SANs, when optimized with φ, produced interpretable kernels across diverse datasets (Physionet, UCI-epilepsy, MNIST, FMNIST).
• Sparse activation functions contributed to reduced model complexity and improved representation efficiency.

Conclusions:

• The φ metric provides an unbiased evaluation of unsupervised model complexity and efficiency.
• Sparsely activated networks offer a promising approach for learning efficient and interpretable representations.
• Optimizing for compression alongside accuracy leads to more meaningful unsupervised feature learning.