Compact and Interpretable Neural Networks Using Lehmer Activation Units.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Conventional neural networks rely on pointwise activation functions that separate feature weighting and nonlinearity.
• Existing activation functions lack intrinsic interpretability and can lead to complex, deep architectures.

Purpose of the Study:

• To introduce Lehmer Activation Units (LAUs) as a novel class of aggregation-based neural activations.
• To unify feature weighting and nonlinearity into a single, differentiable operator.
• To develop both real-valued and complex-valued formulations of LAUs for enhanced expressive capacity.

Main Methods:

• Derivation of LAUs from the Lehmer transform, enabling aggregation-based feature processing.
• Development of learnable parameters within LAUs to adapt aggregation behavior and enhance interpretability.
• Establishment of a universal approximation theorem for LAU-based neural networks.

Main Results:

• LAUs operate on feature collections, offering intrinsically interpretable representations.
• Complex-valued LAUs facilitate phase-sensitive interactions and increased model expressiveness.
• LAU-based networks achieve strong performance with highly compact architectures, concentrating expressive power within neurons.

Conclusions:

• LAUs present a principled, interpretable, and efficient alternative to traditional activation functions.
• The study demonstrates that significant expressive power can be achieved through specialized neuron designs rather than solely architectural depth.
• LAUs offer a promising direction for developing more efficient and understandable deep learning models.