Embeddings hidden layers learning for neural network compression.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Large Language Models (LLMs) present deployment challenges on resource-constrained devices due to their massive parameter counts.
• Effective model compression techniques are crucial for enabling LLMs in edge computing and embedded systems.

Purpose of the Study:

• To introduce a novel parameter-sharing method for neural network compression.
• To develop a new family of efficient neural network architectures (ShareBERT).
• To demonstrate the effectiveness of the proposed method in reducing model size while preserving performance.

Main Methods:

• A new parameter-sharing technique is proposed, leveraging the embedding matrix to learn hidden layers.
• A new architecture family, ShareBERT, is introduced based on this method.
• Evaluations were conducted on multiple linguistic benchmarks across various neural architectures and tasks.

Main Results:

• ShareBERT achieves up to 95.5% of BERT accuracy with only 5 million parameters, a 21.9x reduction.
• The compression method enhances, rather than hinders, representation learning capabilities.
• The approach is robust and flexible across diverse neural network types, layers, and tasks.

Conclusions:

• The proposed parameter-sharing method enables significant model compression, leading to near-zero parameter architectures.
• ShareBERT facilitates efficient deployment of advanced sequence models on low-powered and embedded devices.
• This method is orthogonal to existing compression techniques, offering synergistic potential.