Decentralized Kernel Ridge Regression Based on Data-Dependent Random Feature.

Area of Science:

• Machine Learning
• Decentralized Systems
• Statistical Learning Theory

Background:

• Random features (RFs) are crucial for node consistency in decentralized kernel ridge regression (KRR).
• Existing methods require identical RFs across nodes, limiting adaptability to varied data distributions.
• Significant data variations across nodes necessitate flexible, data-dependent RF generation.

Purpose of the Study:

• To propose a new decentralized KRR algorithm that overcomes limitations of identical RFs.
• To enable adaptive and data-dependent RF generation for improved performance in heterogeneous environments.
• To achieve consensus on decision functions rather than feature representations.

Main Methods:

• Developed a novel decentralized KRR algorithm focusing on consensus of decision functions.
• The proposed method allows for flexible and data-adaptive generation of random features (RFs) on each node.
• Rigorous convergence analysis and numerical verification on multiple real-world datasets were performed.

Main Results:

• The new algorithm demonstrates rigorous convergence properties.
• Achieved an average regression accuracy improvement of 25.5% across six diverse real-world datasets.
• Maintained communication costs comparable to existing methods while significantly enhancing accuracy.

Conclusions:

• The proposed decentralized KRR algorithm effectively handles data heterogeneity by adapting RFs.
• Consensus on decision functions offers greater flexibility and superior performance compared to fixed RF approaches.
• This method provides a more robust and accurate solution for decentralized learning tasks with varied data.