Fast generalization error bound of deep learning without scale invariance of activation functions

Area of Science:

• Deep Learning Theory
• Machine Learning Analysis
• Neural Network Architectures

Background:

• Theoretical analysis of deep learning seeks to identify performance-driving features.
• Suzuki's (2018) generalization error framework suggests scale invariance in activation functions is crucial for tight error bounds.
• Common activation functions like sigmoid, tanh, and ELU lack scale invariance, potentially leading to slower convergence rates (O(1/n)).

Purpose of the Study:

• To derive a tight generalization error bound for deep neural networks with general activation functions, irrespective of scale invariance.
• To investigate whether scale invariance is truly essential for achieving fast convergence rates within Suzuki's (2018) theoretical framework.
• To demonstrate the broader applicability of Suzuki's (2018) framework to diverse activation functions in deep learning.

Main Methods:

• Applied Suzuki's (2018) generalization error analysis framework.
• Derived tight generalization error bounds for deep neural networks utilizing non-scale invariant activation functions.
• Theoretically analyzed convergence rates based on activation function properties.

Main Results:

• A tight generalization error bound was derived for deep neural networks with non-scale invariant activation functions, matching the quality of bounds derived under scale invariance.
• Demonstrated that scale invariance is not a prerequisite for achieving a fast rate of convergence within the analyzed framework.
• The derived bounds are essentially the same as those obtained by Suzuki (2018).

Conclusions:

• Scale invariance of activation functions is not essential for obtaining fast convergence rates in deep learning, contrary to previous interpretations of Suzuki's (2018) work.
• Suzuki's (2018) theoretical framework is applicable to a wider range of activation functions than previously assumed.
• This research broadens the understanding of deep learning generalization and the role of activation functions in theoretical analysis.