设置规范和等价跳过连接:将深度置于深度集中
Lily H Zhang1, Veronica Tozzo2,3, John M Higgins2,3
1Center for Data Science, New York University, New York, NY.
概括
深度集++和集变压器++模型克服了深度神经网络中的梯度问题,用于集预测. 这些改进的架构在各种任务上实现了高性能,包括一个新的单细胞数据集.
科学领域:
- 机器学习 机器学习
- 深度学习 (Deep Learning) 是一种深度学习.
- 集合理论 集合理论
背景情况:
- 变不变的神经网络对于设置预测是有效的.
- 像Deep Sets和Set Transformer这样的现有架构面临着深度网络的挑战,包括消失/爆炸梯度.
- 层规范化可以通过删除预测信息来阻碍Set Transformer中的性能.
研究的目的:
- 为了解决深度变量不变神经网络中的梯度不稳定性和性能退化.
- 为处理集数据引入新的架构改进.
- 开发和验证用于设置预测任务的增强模型.
主要方法:
- 引入了对等差剩余连接的"清洁路径原理".
- 开发了"设定规范" (sn),这是一种专门用于设置的新型规范化技术.
- 构建并评估了深度设置++和设置变压器++模型.
主要成果:
- 新的模型,深度设置++和设置变压器++,实现高深度.
- 在各种任务中,性能与原始架构相当或优于原始架构.
- 介绍了Flow-RBC,这是一个新的单细胞数据集,用于排列不变的预测.
结论:
- "清洁路径原理"和设定规范有效地使得更深,更高性能的变量不变网络成为可能.
- 深度集++和集变压器++代表了集预测的重大进步.
- 新的数据集促进了现实世界的应用,例如单细胞分析等领域.
相关概念视频
Woodward–Hoffmann Selection Rules and Microscopic Reversibility
3.1K
Electrocyclic reactions, cycloadditions, and sigmatropic rearrangements are concerted pericyclic reactions that proceed via a cyclic transition state. These reactions are stereospecific and regioselective. The stereochemistry of the products depends on the symmetry characteristics of the interacting orbitals and the reaction conditions. Accordingly, pericyclic reactions are classified as either symmetry-allowed or symmetry-forbidden. Woodward and Hoffmann presented the selection criteria for...
3.1K
Second Uniqueness Theorem
1.0K
Consider a region consisting of several individual conductors with a definite charge density in the region between these conductors. The second uniqueness theorem states that if the total charge on each conductor and the charge density in the in-between region are known, then the electric field can be uniquely determined.
In contrast, consider that the electric field is non-unique and apply Gauss's law in divergence form in the region between the conductors and the integral form to the...
In contrast, consider that the electric field is non-unique and apply Gauss's law in divergence form in the region between the conductors and the integral form to the...
1.0K
Routh-Hurwitz Criterion II
292
In the application of the Routh-Hurwitz criterion, two specific scenarios can arise that complicate stability analysis.
The first scenario occurs when a singular zero appears in the first column of the Routh table. This situation creates a division by zero issues. To resolve this, a small positive or negative number, denoted as epsilon (∈), is substituted for the zero. The stability analysis proceeds by assuming a sign for ∈. If ∈ is positive, any sign change in the first...
The first scenario occurs when a singular zero appears in the first column of the Routh table. This situation creates a division by zero issues. To resolve this, a small positive or negative number, denoted as epsilon (∈), is substituted for the zero. The stability analysis proceeds by assuming a sign for ∈. If ∈ is positive, any sign change in the first...
292
Routh-Hurwitz Criterion I
273
Consider an electrical power grid, where stability is essential to prevent blackouts. The Routh-Hurwitz criterion is a valuable tool for assessing system stability under varying load conditions or faults. By analyzing the closed-loop transfer function, the Routh-Hurwitz criterion helps determine whether the system remains stable.
To apply the Routh-Hurwitz criterion, a Routh table is constructed. The table's rows are labeled with powers of the complex frequency variable s, starting from the...
To apply the Routh-Hurwitz criterion, a Routh table is constructed. The table's rows are labeled with powers of the complex frequency variable s, starting from the...
273
Divergence and Curl
1.7K
The divergence of a vector field at a point is the net outward flow of the flux out of a small volume through a closed surface enclosing the volume, as the volume tends to zero. More practically, divergence measures how much a vector field spreads out or diverges from a given point. For an outgoing flux, conventionally, the divergence is positive. The diverging point is often called the "source" of the field. Meanwhile, the negative divergence of a vector field at a point means that the...
1.7K
Castigliano's Theorem
433
Castigliano's theorem analyzes displacements and rotations in elastic structures. It relates the derivative of elastic strain energy to the applied forces or moments, allowing for the calculation of deformations. The theorem states that the partial derivative of the total strain energy of a system with respect to a specific load results in the displacement at the point where the load is applied. This principle applies to both forces and moments.
433


