OOPS: Outlier-aware and quadratic programming based structured pruning for large language models.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Large Language Models (LLMs) face deployment challenges due to their substantial size and resource demands.
• Structured pruning methods aim to mitigate these issues, categorized into retraining-free and retraining-based approaches.
• Existing retraining-free methods often lead to performance degradation, while retraining-based methods are computationally intensive.

Purpose of the Study:

• To develop an efficient structured pruning framework for Large Language Models (LLMs) that minimizes resource consumption and maintains performance.
• To address the limitations of existing retraining-free and retraining-based pruning techniques.
• To propose a novel framework, OOPS (Outlier-aware and quadratic prOgramming based Structured Pruning), that balances efficiency and effectiveness.

Main Methods:

• OOPS employs a three-component framework: outlier-aware pruning unit selection, quadratic programming-based reconstruction, and layer-wise distillation.
• The outlier-aware unit selection and quadratic programming reconstruction enable retraining-free pruning.
• Layer-wise distillation is optionally incorporated to further enhance performance, particularly for retraining-based scenarios.

Main Results:

• OOPS demonstrates superior performance compared to existing retraining-free methods without requiring retraining.
• When incorporating layer-wise distillation, OOPS outperforms retraining-based methods with reduced computational costs.
• Evaluations across 11 models from 4 LLM families and multiple tasks confirm OOPS's effectiveness.

Conclusions:

• OOPS offers an effective and efficient solution for structured pruning of Large Language Models.
• The framework successfully reduces model size and resource consumption while preserving or enhancing performance.
• OOPS provides a versatile approach, excelling in both retraining-free and retraining-based pruning paradigms.