Learning of Gaussian Processes in Distributed and Communication Limited Systems

Area of Science:

• Machine Learning
• Distributed Systems
• Information Theory

Background:

• Distributed and communication-limited systems require efficient algorithms for optimal statistical model learning.
• Gaussian Processes (GP) are a key example for studying optimal distributed learning strategies.
• Estimating inner-products of Gaussian vectors is a fundamental challenge in these systems.

Purpose of the Study:

• To characterize optimal strategies for distributed learning in communication-limited environments.
• To determine the minimum bits required for estimating Gaussian vector inner-products across distributed machines.
• To develop and evaluate practical distributed GP learning methods.

Main Methods:

• Utilizing information-theoretic bounds to derive an optimal solution for inner-product estimation via vector quantization.
• Proposing two suboptimal, practical schemes: vector quantization and per-symbol quantization.
• Integrating these inner-product calculation schemes into distributed GP learning algorithms.

Main Results:

• An optimal solution for Gaussian vector inner-product estimation using vector quantization was established.
• Per-symbol quantization demonstrated near-optimal performance, offering a practical alternative.
• The proposed distributed GP learning methods, using the developed communication schemes, outperformed existing zero-rate methods like BCM and PoE.

Conclusions:

• Efficient inner-product estimation is crucial for optimal distributed GP learning.
• Practical schemes like per-symbol quantization provide a strong balance between performance and communication cost.
• The developed methods enable effective distributed GP learning even with minimal bit expenditure per symbol.