Bayesian Neural Networks with Weight Sharing Using Dirichlet Processes

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computational Neuroscience

Background:

• Feed-forward neural networks (FFNNs) often contain a large number of parameters, leading to high memory requirements and computational costs.
• Existing weight sharing methods can be rigid and may not optimally adapt to specific data characteristics.
• Efficient parameter reduction in deep learning models is crucial for deployment on resource-constrained devices.

Purpose of the Study:

• To develop a novel method for automatic and data-adaptive weight sharing in feed-forward neural networks.
• To significantly reduce the number of parameters and memory footprint of neural networks.
• To improve the performance of neural networks through efficient parameter sharing.

Main Methods:

• Incorporation of a Dirichlet process prior over the weight distribution in FFNNs.
• Alternating sampling from the posterior of weights and the posterior of assignment of network connections to weights.
• Development of computational techniques to reduce the burden of the sampling procedure.

Main Results:

• The proposed model achieved significant reductions in the number of network parameters and memory footprint.
• Experimental results demonstrated superior performance compared to models employing random weight sharing.
• The data-adaptive weight sharing mechanism proved effective in optimizing network efficiency.

Conclusions:

• The Dirichlet process prior offers an effective approach for data-adaptive weight sharing in FFNNs.
• This method substantially reduces model size and memory requirements while maintaining or improving performance.
• The approach presents a promising direction for developing more efficient and scalable deep learning models.