Hybrid-DCA: A double asynchronous approach for stochastic dual coordinate ascent

Area of Science:

• High-performance computing
• Distributed machine learning
• Optimization algorithms

Background:

• Stochastic dual coordinate ascent (SDCA) has been parallelized using shared-memory or distributed-memory approaches.
• Existing methods face limitations with large datasets and common multi-core cluster architectures.

Purpose of the Study:

• To propose a hybrid SDCA framework for multi-core clusters, combining shared and distributed memory paradigms.
• To develop a double asynchronous method for efficient large-scale machine learning.

Main Methods:

• Data distribution across nodes in a multi-core cluster.
• Asynchronous parallel local problem solving on each node's cores.
• Asynchronous across-node update scheme for aggregating local solutions.

Main Results:

• The hybrid SDCA framework converges to a global solution for L-Lipschitz continuous loss functions.
• Achieves linear convergence rate for smooth convex loss functions.
• Demonstrates superior scalability and speed compared to shared-memory and distributed-memory methods.

Conclusions:

• The proposed double asynchronous hybrid SDCA is highly effective for large-scale datasets.
• Significantly outperforms existing distributed algorithms like CoCoA+ in terms of speed and scalability.
• Enables solving massive datasets (e.g., 280 GB) in under 30 seconds on a 16-node cluster.