Efficient self-attention with smart pruning for sustainable large language models.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Large Language Models (LLMs) offer advanced multitasking capabilities but suffer from high computational demands, leading to significant environmental concerns regarding energy and water consumption.
• The internal transformer layers are primary contributors to the computational complexity and resource intensity of LLMs.

Purpose of the Study:

• To propose and evaluate an innovative compression approach for reducing the size and computational footprint of Large Language Models (LLMs).
• To address the environmental impact associated with the high energy and water consumption of LLMs through efficient model compression.

Main Methods:

• The proposed method combines mathematical and structural techniques for model compression, focusing on transformer layers.
• Forward Propagation Pruning (FPP) is employed to compress embedding and feed-forward layers using weight freezing and zeroing.
• Weight Matrix Folding, incorporating Identical Row Compression (IRC) and Diagonal Weight Compression (DWC), is used to prune self-attention layer matrices.

Main Results:

• The compression approach achieved a 99% compression of transformer layers and 70% for linear layers, leading to an overall model compression of approximately 70%.
• Model accuracy was maintained at nearly original levels post-compression.
• Moderate compression rates (20-40%) demonstrated stable or improved model performance, alongside significant reductions in memory usage and computational demands.

Conclusions:

• The developed compression technique effectively reduces LLM size and resource requirements, making them more energy-efficient.
• This approach offers a viable path towards more sustainable AI development and deployment.
• Optimized LLMs can achieve performance benefits while minimizing environmental impact.