GradFreeBits: Gradient-Free Bit Allocation for Mixed-Precision Neural Networks.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Quantized Neural Networks (QNNs) are crucial for deploying deep learning models on resource-constrained edge devices.
• Mixed-precision quantization offers improved performance-computation trade-offs but presents optimization challenges due to discrete bit allocations and inter-layer dependencies.

Purpose of the Study:

• To develop a novel optimization scheme for training mixed-precision QNNs that addresses the difficulties of optimizing discrete bit allocations.
• To improve the efficiency and performance of QNNs on edge devices through effective mixed-precision training.

Main Methods:

• Proposes GradFreeBits, a joint optimization scheme alternating between gradient-based weight optimization and gradient-free bit allocation optimization.
• Applies the method to optimize precision levels across different layers of QNNs, considering inter-layer dependencies.

Main Results:

• Achieves state-of-the-art or comparable performance on image classification (CIFAR10/100, ImageNet) and semantic segmentation (Cityscapes) tasks.
• Demonstrates effectiveness across various neural network architectures, including graph neural networks.

Conclusions:

• GradFreeBits offers an effective solution for training mixed-precision QNNs, overcoming traditional optimization hurdles.
• The proposed method is versatile and extensible to other neural network applications with challenging parameter optimization requirements.