从ReLU到GeMU:在圆投影镜头中的激活功能
Jiayun Li1, Yuxiao Cheng1, Yiwen Lu1
1Department of Automation and BNRist, Tsinghua University, Beijing, 100084, China.
概括
研究人员将修正线性单元 (ReLU) 激活函数概括为通用多变量投影单元 (GeMU). 在凸上投射的GeMU,为神经网络提供了比ReLU更大的表达力.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 深度学习 (Deep Learning) 是一种深度学习.
背景情况:
- 激活函数为神经网络引入非线性.
- 修正线性单位 (ReLU) 是一个广泛使用的激活函数,因为它的简单性和有效性.
- 送神经网络 (FNN) 与预测梯度下降 (PGD) 算法具有结构上的相似之处.
研究的目的:
- 通过将其解释为投影来概括ReLU激活函数.
- 引入一个新的激活函数,即通用多变量投影单元 (GeMU).
- 调查使用GeMU的FNN的表达力和性能.
主要方法:
- 将ReLU解释为从R投射到非负半线R+上的投影.
- 将ReLU概括为GeMU,这是一个投影运算器在凸上,就像第二阶 (SOC) 这样.
- 证明GeMU激活的FNN与ReLU激活的FNN相比具有更强的表达力.
- 在各种神经网络架构和任务中进行实验性评估.
主要成果:
- 由GeMU激活的FNN的表达力严格地大于由ReLU激活的.
- GeMU在不同的神经网络架构中展示了多功能性.
- 在各种任务中,GeMU的性能优于现有的激活功能.
- GeMU为增强神经网络能力提供了一种新的方法.
结论:
- 作为一个激活功能,GeMU代表了ReLU的显著进步.
- 激活函数的基于投影的泛化为研究开辟了新的途径.
- GeMU为深度学习模型提供了更好的性能和多功能性.
相关概念视频
01:20Convolution Properties I
142
Convolution computations can be simplified by utilizing their inherent properties.
The commutative property reveals that the input and the impulse response of an LTI (Linear Time-Invariant) system can be interchanged without affecting the output:
The commutative property reveals that the input and the impulse response of an LTI (Linear Time-Invariant) system can be interchanged without affecting the output:
142
01:17Convolution Properties II
177
The important convolution properties include width, area, differentiation, and integration properties.
The width property indicates that if the durations of input signals are T1 and T2, then the width of the output response equals the sum of both durations, irrespective of the shapes of the two functions. For instance, convolving two rectangular pulses with durations of 2 seconds and 1 second results in a function with a width of 3 seconds.
The area property asserts that the area under the...
The width property indicates that if the durations of input signals are T1 and T2, then the width of the output response equals the sum of both durations, irrespective of the shapes of the two functions. For instance, convolving two rectangular pulses with durations of 2 seconds and 1 second results in a function with a width of 3 seconds.
The area property asserts that the area under the...
177
01:24Convolution: Math, Graphics, and Discrete Signals
237
In any LTI (Linear Time-Invariant) system, the convolution of two signals is denoted using a convolution operator, assuming all initial conditions are zero. The convolution integral can be divided into two parts: the zero-input or natural response and the zero-state or forced response, with t0 indicating the initial time.
To simplify the convolution integral, it is assumed that both the input signal and impulse response are zero for negative time values. The graphical convolution process...
To simplify the convolution integral, it is assumed that both the input signal and impulse response are zero for negative time values. The graphical convolution process...
237
01:16Vector Representation of Complex Numbers
117
Complex numbers, represented in Cartesian coordinates, can also be visualized as vectors. These vectors can be expressed in polar form, emphasizing their magnitude and angle. When a complex number is input into a function, the output is another complex number, highlighting the function's zero point from which the vector representation can originate.
Consider a function defined as the product of the complex factors in the numerator divided by the product of the complex factors in the...
Consider a function defined as the product of the complex factors in the numerator divided by the product of the complex factors in the...
117
01:14Gradient and Del Operator
2.5K
In mathematics and physics, the gradient and del operator are fundamental concepts used to describe the behavior of functions and fields in space. The gradient is a mathematical operator that gives both the magnitude and direction of the maximum spatial rate of change. Consider a person standing on a mountain. The slope of the mountain at any given point is not defined unless it is quantified in a particular direction. For this reason, a "directional derivative" is defined, which is a vector...
2.5K
01:29Coordinates and Map Projections
36
Coordinates and map projections are essential tools in accurately representing the Earth's surface for various applications, ranging from navigation to spatial analysis. The latitude and longitude coordinate system is a universally recognized framework for defining locations. Latitude specifies the distance of a point north or south of the equator, measured in degrees from 0° at the equator to 90° at the poles. Longitude indicates a location's position east or west of the prime meridian,...
36