Augmenting interaction effects in convolutional networks with taylor polynomial gated units.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Transformer-based vision models are often favored for their ability to capture long-range dependencies.
• However, recent convolutional neural network (CNN) models like ConvNeXt challenge the notion of Transformer superiority.
• This suggests alternative methods for modeling input interactions in CNNs are valuable.

Purpose of the Study:

• To explore efficient methods for modeling input interactions in CNNs without relying on attention mechanisms.
• To introduce a novel activation function that captures interaction effects within convolutional architectures.
• To investigate the potential of polynomial-based functions in modern neural networks.

Main Methods:

• Proposed the Taylor Polynomial Gated Unit (TPGU) as a novel activation function.
• TPGU replaces the cumulative distribution function in GELU with a learnable Taylor polynomial.
• Evaluated TPGU's performance by replacing GELU in ConvNeXt models under identical training conditions.

Main Results:

• Swapping GELU with TPGU activation functions significantly improved model performance.
• TPGU demonstrated a particularly favorable impact on pure convolutional networks.
• ConvNeXt-T performance was enhanced by 0.7% on the ImageNet-1K dataset using TPGU.

Conclusions:

• The Taylor Polynomial Gated Unit (TPGU) offers an effective alternative to existing activation functions for CNNs.
• Polynomial-based functions show promise for improving contemporary neural network architectures.
• TPGU provides a method to efficiently model input interactions without attention, boosting performance in convolutional models.