Towards efficient language giants: A comprehensive survey on structural optimizations and compression techniques for...

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Large language models (LLMs) demonstrate significant success in Natural Language Processing (NLP) tasks.
• However, LLMs present substantial computational and memory challenges for deployment.
• Efficient deployment is crucial for latency-sensitive and resource-constrained applications.

Purpose of the Study:

• To provide a comprehensive overview of model-level optimization techniques for LLMs.
• To address the computational and memory overhead associated with LLM inference.
• To guide practical strategies for efficient LLM deployment.

Main Methods:

• Examined efficient module designs for core Transformer components (attention, feed-forward networks, normalization, positional encoding).
• Summarized compression methods including quantization, pruning, knowledge distillation, approximation, and parameter sharing.
• Categorized recent advances, analyzing algorithmic principles, trade-offs, and inference efficiency impact.

Main Results:

• Identified key model-level optimization techniques for LLMs.
• Analyzed the effectiveness of various compression and efficient module design strategies.
• Demonstrated significant reductions in inference costs through combined approaches.

Conclusions:

• Efficient module designs and compression techniques are vital for overcoming LLM deployment challenges.
• Combining these strategies offers practical solutions for reducing inference costs.
• Optimized LLMs can meet specific accuracy and latency requirements for diverse applications.