相关实验视频
Updated: Sep 9, 2025

13:19
Deep Neural Networks for Image-Based Dietary Assessment
Published on: March 13, 2021
9.3K
mL-BFGS:用于分布式大规模神经网络优化的基于动量的L-BFGS
Yue Niu1, Zalan Fabian1, Sunwoo Lee2
1Department of Electrical and Computer Engineering, University of Southern California.
概括
我们介绍了mL-BFGS, 一种基于动量的算法, 这种方法稳定了融合,并加速了大规模分布式模型的训练.
科学领域:
- 机器学习
- 优化算法
- 深度神经网络
背景情况:
- 准牛顿方法,包括L-BFGS,由于计算成本和随机设置的不稳定性,在大规模深度神经网络训练中面临挑战.
- 现有的L-BFGS适用于随机训练通常会带来显著的开销,从而抵消了趋同的好处.
研究的目的:
- 提出mL-BFGS,一个轻量级的,基于动量的L-BFGS算法,旨在高效大规模分布式深度神经网络优化.
- 在深度学习中提高准牛顿方法的稳定性和减少计算负担.
主要方法:
- 开发了mL-BFGS,将动量方案纳入L-BFGS更新中,以减轻赫森近似中的随机噪声.
- 在mL-BFGS中实现区块智能赫斯近似,以在大型训练中跨节点分配计算和内存成本.
- 在随机优化场景中提供了mL-BFGS的理论收分析.
主要成果:
- 在随机优化过程中,mL-BFGS通过降低赫斯近似的噪声来证明稳定收.
- 区块智能赫斯近似使得分布式训练能够有效地扩展计算和内存.
- 与SGD和Adam等基线方法相比,对基准神经模型的实证结果显示了显著的代和壁表加快.
结论:
- 在大型深度神经网络训练中,mL-BFGS提供了一种有前途的方法.
- 拟议的算法有效地平衡了计算效率与收稳定性,优于现有方法.
相关概念视频
Distributed Loads: Problem Solving
729
Beams are structural elements commonly employed in engineering applications requiring different load-carrying capacities. The first step in analyzing a beam under a distributed load is to simplify the problem by dividing the load into smaller regions, which allows one to consider each region separately and calculate the magnitude of the equivalent resultant load acting on each portion of the beam. The magnitude of the equivalent resultant load for each region can be determined by calculating...
729
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
100
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
100
Maxwell-Boltzmann Distribution: Problem Solving
1.7K
Individual molecules in a gas move in random directions, but a gas containing numerous molecules has a predictable distribution of molecular speeds, which is known as the Maxwell-Boltzmann distribution, f(v).
This distribution function f(v) is defined by saying that the expected number N (v1,v2) of particles with speeds between v1 and v2 is given by
This distribution function f(v) is defined by saying that the expected number N (v1,v2) of particles with speeds between v1 and v2 is given by
1.7K
Distributed Loads
605
Distributed loads are a common type of load that engineers and scientists encounter in various practical situations. Distributed loads often refer to a type of load spread over a surface or a structure and can be modeled as continuous force per unit area.
For example, consider a bookshelf filled with books stacked vertically adjacent to each other. The weight of the books is evenly distributed over the length of the shelf. As a result, the pressure at different locations on the surface of the...
For example, consider a bookshelf filled with books stacked vertically adjacent to each other. The weight of the books is evenly distributed over the length of the shelf. As a result, the pressure at different locations on the surface of the...
605
Multimachine Stability
227
Multimachine stability analysis is crucial for understanding the dynamics and stability of power systems with multiple synchronous machines. The objective is to solve the swing equations for a network of M machines connected to an N-bus power system.
In analyzing the system, the nodal equations represent the relationship between bus voltages, machine voltages, and machine currents. The nodal equation is given by:
In analyzing the system, the nodal equations represent the relationship between bus voltages, machine voltages, and machine currents. The nodal equation is given by:
227
Linear Momentum in Control Volume
1.1K
Newton's second law is applied to obtain the linear momentum in a control volume in a fluid system. According to this law, the rate of change of linear momentum is equal to the sum of external forces acting on the system. When a control volume matches the fluid system at a specific moment, the forces acting on both are identical. Reynolds transport theorem helps explain this by breaking down the system's linear momentum into two components: the rate of change of linear momentum within...
1.1K

