Generating Accurate Pseudo-labels in Semi-Supervised Learning and Avoiding Overconfident Predictions via Hermite...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Rectified Linear Units (ReLUs) are prevalent activation functions in computer vision.
• Theoretical work suggests basis expansions, like polynomials, may outperform ReLUs in optimization and generalization.
• Existing research is limited to shallow networks, with practical viability unconfirmed.

Purpose of the Study:

• To investigate Hermite polynomial expansions as a substitute for ReLUs in deep neural networks.
• To evaluate the performance of Hermite activations in supervised and semi-supervised learning settings.
• To analyze the theoretical properties and practical benefits of Hermite activation functions.

Main Methods:

• Implemented deep neural networks utilizing Hermite polynomial expansions as activation functions.
• Conducted experiments in both supervised and semi-supervised learning (SSL) / transductive learning scenarios.
• Performed theoretical analysis to assess network robustness and mathematical properties.

Main Results:

• Hermite polynomial activations demonstrated significant benefits in semi-supervised learning (SSL) settings.
• The use of Hermite activations led to improved pseudo-label accuracies.
• Sizable financial savings were observed due to concurrent runtime benefits.
• Theoretical analysis indicated enhanced robustness to noise and other favorable mathematical properties.

Conclusions:

• Hermite polynomial activations offer a viable and beneficial alternative to ReLUs, particularly in semi-supervised learning.
• This approach yields practical advantages including improved accuracy and computational efficiency.
• Networks with Hermite activations possess desirable theoretical properties like noise robustness.