通信效率高的分布式立方牛顿与压缩的惰的赫西安.
Zhen Zhang1, Keqin Che1, Shaofu Yang1
1School of Computer Science and Engineering, Southeast University, Nanjing, Jiangsu, PR China.
概括
我们开发了使用立方牛顿方法和压缩惰的Hessians的通信效率高的二级分布式优化算法. 这些方法大大降低了分布式学习的通信成本,同时保持了快速的融合率.
科学领域:
- 分布式机器学习 分布式机器学习
- 优化算法的优化算法
背景情况:
- 第二阶级的分布式优化算法比第一阶级的方法提供更快的融合.
- 一个主要的挑战是二级算法固有的通信瓶.
研究的目的:
- 在参数-服务器框架内提出通信效率高的二级分布式优化算法.
- 解决通信瓶,同时保留二级方法的好处.
主要方法:
- 合并立方牛顿方法与压缩惰的赫西安人.
- 工作人员选择性地与服务器通信压缩的hessian,减少通信开销.
- 梯度规范化的应用在凸问题上.
主要成果:
- 与最先进的方法相比,理论证明了与非凸问题的通信成本的降低.
- 对于非凸问题,保留了O{\displaystyle O} - 3/2的代复杂度,与集中立方牛顿方法相匹配.
- 在凸问题上实现了全球收,在强烈凸问题上实现了局部超线性收.
结论:
- 拟议的算法有效地降低了分布式学习中的通信成本.
- 它们保持了具有竞争力的代复杂性,并为各种问题类型实现了强大的融合保证.
- 数字实验验证了开发的算法的高效率.
相关概念视频
Area Computation by the Alternative Coordinate Method
53
The alternative coordinate method, also known as the Shoelace Formula, is a technique for determining the area of a traverse using Cartesian coordinates. This method relies on the sequential arrangement of x and y coordinates for each point of the shape, ensuring accuracy and ease of application.In this approach, each corner's x and y coordinates are listed as fractions, with the x-coordinate as the numerator and the y-coordinate as the denominator. These coordinates are arranged sequentially...
53
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
53
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
53
Distributed Loads: Problem Solving
645
Beams are structural elements commonly employed in engineering applications requiring different load-carrying capacities. The first step in analyzing a beam under a distributed load is to simplify the problem by dividing the load into smaller regions, which allows one to consider each region separately and calculate the magnitude of the equivalent resultant load acting on each portion of the beam. The magnitude of the equivalent resultant load for each region can be determined by calculating...
645
Fast Decoupled and DC Powerflow
191
The fast decoupled power flow method addresses contingencies in power system operations, such as generator outages or transmission line failures. This method provides quick power flow solutions, essential for real-time system adjustments. Fast decoupled power flow algorithms simplify the Jacobian matrix by neglecting certain elements, leading to two sets of decoupled equations:
191
Cartesian Form for Vector Formulation
634
The Cartesian form for vector formulation is a process to calculate the moment of force using the position and force vectors. The moment of force is defined as the cross-product of these vectors, making it a vector quantity. The Cartesian form of the position and force vectors involves unit vectors, which can be used to express the cross-product in determinant form.
634
Gauss's Law: Problem-Solving
1.7K
Gauss's law helps determine electric fields even though the law is not directly about electric fields but electric flux. In situations with certain symmetries (spherical, cylindrical, or planar) in the charge distribution, the electric field can be deduced based on the knowledge of the electric flux. In these systems, we can find a Gaussian surface S over which the electric field has a constant magnitude. Furthermore, suppose the electric field is parallel (or antiparallel) to the area...
1.7K


