Coding for Large-Scale Distributed Machine Learning

Area of Science:

• Computer Science
• Information Theory
• Machine Learning

Background:

• Machine learning is increasingly distributed due to large data volumes and sensor networks.
• Large-scale distributed machine learning (DML) faces challenges like delay, errors, and efficiency.
• Existing solutions include error-control and performance-boosting schemes, with recent focus on error-control coding.

Purpose of the Study:

• To provide a comprehensive review of coding principles and recent developments for large-scale DML.
• To introduce theories and algorithms for applying coding to DML systems.
• To address the lack of surveys on coding for large-scale learning.

Main Methods:

• Review of gradient coding with optimal code distance for primal-based DML.
• Introduction to random coding for gradient-based DML.
• Proposal of a separate coding method for the two steps in primal-dual-based DML (ADMM).

Main Results:

• Coding offers benefits for DML, including high efficiency and low complexity.
• Specific coding schemes are discussed for different steps in distributed optimization (primal-based and primal-dual-based).
• The review highlights the potential of error-control coding to improve DML reliability.

Conclusions:

• Coding is a promising approach to tackle challenges in large-scale DML.
• Further research directions are identified for advancing coding techniques in DML.
• This work provides a foundational understanding for future developments in efficient and reliable DML.