Toward Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Graph Neural Networks (GNNs) face challenges like over-smoothing and vanishing gradients.
• Graph-Augmented MLP (GA-MLP) models improve accuracy but suffer from efficiency degradation.
• Existing acceleration methods are ineffective for GA-MLP due to graph data dependencies.

Purpose of the Study:

• To propose a novel parallel deep learning framework for GA-MLP models.
• To enhance the efficiency and scalability of GA-MLP models.
• To address the limitations of existing acceleration techniques in graph deep learning.

Main Methods:

• Introduced a parallel graph deep learning Alternating Direction Method of Multipliers (pdADMM-G) framework for model parallelism.
• Developed the extended pdADMM-G-Q algorithm incorporating quantization to reduce communication costs.
• Provided theoretical convergence analysis with a sublinear rate of o(1/k).

Main Results:

• Demonstrated the convergence of both pdADMM-G and pdADMM-G-Q algorithms through extensive experiments.
• Achieved significant speedups and superior performance compared to state-of-the-art methods on nine benchmark datasets.
• The pdADMM-G-Q algorithm reduced communication overheads by up to 45% without performance loss.

Conclusions:

• The proposed pdADMM-G framework effectively addresses the efficiency challenges of GA-MLP models.
• The pdADMM-G-Q algorithm offers a practical solution for reducing communication costs in parallel graph deep learning.
• These advancements enable more scalable and efficient deep learning on graph data.