局部内核重规范化作为超参数化的卷积神经网络中特征学习的机制
R Aiudi1,2, R Pacelli3, P Baglioni4
1Dipartimento di Scienze Matematiche, Fisiche e Informatiche, Università degli Studi di Parma, Parma, Italy.
Nature communications
|January 10, 2025
概括
卷积神经网络 (CNN) 由于局部内核重规范化,在有限宽度设置中表现出优异的性能,与完全连接的网络不同. 这种机制能够在浅层的CNN中实现特征学习,这是完全连接架构中缺少的功能.
科学领域:
- 机器学习 机器学习
- 计算机视觉 计算机视觉
- 理论神经科学 理论神经科学
背景情况:
- 完全连接的神经网络 (无限宽度限制) 在计算机视觉中往往优于有限宽度的同行.
- 现代卷积神经网络 (CNN) 架构在有限宽的模式中表现出色.
研究的目的:
- 提供一个理论框架,解释有限和无限宽度神经网络之间的性能差异.
- 分析卷积层与浅层网络中完全连接层的行为.
主要方法:
- 在一个隐藏层卷积网络的比例极限中导出有效作用.
- 对完全连接网络的理论结果进行比较.
主要成果:
- 与完全连接的网络相比,在卷积网络中确定了一种独特的内核重新规范化形式.
- 卷积内核经历了局部重新规范化,使得数据依赖的预测组件可以被选择.
- 完全连接的网络内核只经历全球性重新规范化.
结论:
- 在CNN中,本地内核的重新规范化为过度参数化的浅层网络中的特征学习提供了一种机制.
- 这种功能学习能力是CNN的特点,在没有重量共享的浅层完全连接或局部连接网络中并不存在.
相关概念视频
Linear Approximation in Frequency Domain
85
Linear systems are characterized by two main properties: superposition and homogeneity. Superposition allows the response to multiple inputs to be the sum of the responses to each individual input. Homogeneity ensures that scaling an input by a scalar results in the response being scaled by the same scalar.
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear....
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear....
85
Reducing Line Loss
143
In a three-phase circuit, line loss is an indicator of energy dissipated as heat due to the resistance of transmission lines. To address this, incorporating transformers into the system—a step-up transformer at the source and a step-down transformer at the load—is a strategic solution. Two three-phase transformers are introduced to improve this.
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss...
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss...
143
Improving Translational Accuracy
8.7K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
8.7K
Convolution Properties I
136
Convolution computations can be simplified by utilizing their inherent properties.
The commutative property reveals that the input and the impulse response of an LTI (Linear Time-Invariant) system can be interchanged without affecting the output:
The commutative property reveals that the input and the impulse response of an LTI (Linear Time-Invariant) system can be interchanged without affecting the output:
136
Downsampling
127
When considering a sampled sequence with zero values between sampling instants, one can replace it by taking every N-th value of the sequence. At these integer multiples of N, the original and sampled sequences coincide. This process, known as decimation, involves extracting every N-th sample from a sequence, thereby creating a more efficient sequence.
The Fourier transform of the decimated sequence reveals a combination of scaled and shifted versions of the original spectrum. This...
The Fourier transform of the decimated sequence reveals a combination of scaled and shifted versions of the original spectrum. This...
127
Convolution Properties II
168
The important convolution properties include width, area, differentiation, and integration properties.
The width property indicates that if the durations of input signals are T1 and T2, then the width of the output response equals the sum of both durations, irrespective of the shapes of the two functions. For instance, convolving two rectangular pulses with durations of 2 seconds and 1 second results in a function with a width of 3 seconds.
The area property asserts that the area under the...
The width property indicates that if the durations of input signals are T1 and T2, then the width of the output response equals the sum of both durations, irrespective of the shapes of the two functions. For instance, convolving two rectangular pulses with durations of 2 seconds and 1 second results in a function with a width of 3 seconds.
The area property asserts that the area under the...
168


