Survey on Efficient Large Language Models: Principles, Algorithms, Applications, and Open Issues.

Area of Science:

• Artificial Intelligence
• Computer Science

Background:

• Large language models (LLMs) face computational and deployment challenges due to their increasing size and complexity.
• Inference optimization techniques are crucial for accelerating LLM performance while maintaining accuracy.

Purpose of the Study:

• To provide a comprehensive survey of LLM inference acceleration strategies.
• To analyze these techniques from foundational, algorithmic, and application-based perspectives.
• To offer insights for researchers and practitioners on scalable and efficient LLM systems.

Main Methods:

• Categorization of existing LLM inference optimization techniques into a new taxonomy.
• Analysis of methods including quantization, pruning, distillation, efficient architectures, compilation, and hardware-aware approaches.
• Examination of technique interactions throughout the LLM lifecycle (training, fine-tuning, serving).

Main Results:

• A structured overview of diverse LLM inference acceleration strategies.
• Identification of key applications and emerging trends in efficient LLM development.
• Discussion of open research challenges and practical considerations for deployment.

Conclusions:

• LLM inference optimization is vital for addressing computational costs and deployment barriers.
• A systematic approach to understanding and applying these techniques is necessary for efficient LLM systems.
• Further research is needed to tackle emerging trends and unresolved issues in the field.