Neural networks with low-resolution parameters.

Area of Science:

• Artificial Intelligence
• Computer Science
• Machine Learning

Background:

• Large neural network models face challenges in memory usage and computational efficiency.
• Optimizing model size is crucial for practical implementation across diverse applications.

Purpose of the Study:

• To investigate the impact of parameter bit precision on neural network model performance.
• To compare low-bit precision models against standard 32-bit models for multiclass object classification.

Main Methods:

• Analysis of models with varying weight resolutions (1-bit to 4.08-bit).
• Inclusion of models with fully connected, convolutional, and transformer layers.
• Evaluation of performance, training epochs, and stability with data augmentation.

Main Results:

• Low-bit precision models (e.g., 2.32-bit) achieve performance comparable to 32-bit models.
• Large parameter models maintain similar performance with fewer bits; smaller models require more training.
• Including zero in weights and careful data augmentation improves stability in low-resolution models.

Conclusions:

• 2.32-bit weights offer an optimal balance for memory reduction, performance, and efficiency.
• Low-bit precision models show promise for memory-constrained devices.
• Further research on diverse datasets and larger models, alongside hardware advancements, is recommended.