Padé Neurons for Efficient Neural Models

Neural networks commonly employ the McCulloch-Pitts neuron model, which is a linear model followed by a point-wise non-linear activation. Various researchers have already advanced inherently non-linear neuron models, such as quadratic neurons, generalized operational neurons, generative neurons, and super neurons, which offer stronger non-linearity compared to point-wise activation functions. In this paper, we introduce a novel and better non-linear neuron model called Padé neurons ( $\mathrm {\textit {Paon}}$ s), inspired by Padé approximants. $\mathrm {\textit {Paon}}$ s offer several advantages, such as diversity of non-linearity, since each $\mathrm {\textit {Paon}}$ learns a different non-linear function of its inputs, and layer efficiency, since $\mathrm {\textit {Paon}}$ s provide stronger non-linearity in much fewer layers compared to piecewise linear approximation. Furthermore, $\mathrm {\textit {Paon}}$ s include all previously proposed neuron models as special cases, thus any neuron model in any network can be replaced by $\mathrm {\textit {Paon}}$ s. We note that there has been a proposal to employ the Padé approximation as a generalized point-wise activation function, which is fundamentally different from our model. To validate the efficacy of $\mathrm {\textit {Paon}}$ s, in our experiments, we replace classic neurons in some well-known neural image super-resolution, compression, and classification models based on the ResNet architecture with $\mathrm {\textit {Paon}}$ s. Our comprehensive experimental results and analyses demonstrate that neural models built by $\mathrm {\textit {Paon}}$ s provide better or equal performance than their classic counterparts with a smaller number of layers. The PyTorch implementation code for $\mathrm {\textit {Paon}}$ is open-sourced at https://github.com/onur-keles/Paon.