Bitscaling: Streamlining neural network compression via predictive multi-scale growth of mixed-precision networks

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Jointly optimizing neural network scale and quantization offers superior compression but faces a vast search space.
• Existing methods struggle with practicality due to the combinatorial complexity of co-compression.

Purpose of the Study:

• To develop a unified framework, BitScaling, for efficient co-compression of neural networks.
• To extend compute scaling laws to mixed-precision quantized networks for better prediction of optimal configurations.

Main Methods:

• Extended compute scaling law to a unified continuous function for test loss, model scale, and bit-width under a fixed budget (Billion Operations Per Second - BOPs).
• Introduced BitScaling, a co-compression framework utilizing dimension-reduced supernet proxies for efficient mixed-precision quantization search.
• Incorporated quantization-aware model growth for additional data fitting and prediction of optimal scaling ratios and bit allocations.

Main Results:

• BitScaling achieved search speeds up to 6.82 times faster than existing mixed-precision quantization (MPQ) methods.
• Demonstrated up to 85.77% reduction in memory usage compared to traditional MPQ approaches.
• Matched or surpassed state-of-the-art co-compression methods in top-1 accuracy under extreme low-budget constraints on ImageNet.

Conclusions:

• BitScaling provides an efficient and effective solution for neural network co-compression.
• The framework enables a superior trade-off between compression efficiency and model performance, especially in resource-constrained environments.