LS-PRISM: A layer-selective pruning method via low-rank approximation and sparsification for efficient large language...

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Large Language Models (LLMs) achieve state-of-the-art performance in Natural Language Processing (NLP).
• The substantial parameter count of LLMs poses deployment challenges for resource-constrained environments.
• Existing compression methods often apply uniform compression across all layers, potentially impacting performance unevenly.

Purpose of the Study:

• To introduce LS-PRISM, a novel Layer-Selective Pruning via low-Rank Approximation and Sparsification Method.
• To efficiently compress LLMs while preserving performance on critical NLP benchmarks.
• To provide a scalable solution for LLM deployment in resource-limited settings.

Main Methods:

• LS-PRISM employs layer-selective low-rank approximation based on accuracy and loss impact.
• Dynamic Rank Selection adaptively determines approximation ranks for optimal performance retention.
• Unstructured pruning and optional LoRA fine-tuning further enhance model sparsification and performance recovery.

Main Results:

• Significant reductions in parameter count and storage achieved.
• Minimal degradation in accuracy observed across NLP benchmarks (BoolQ, RTE, ARC-Challenge).
• Up to 12% parameter reduction demonstrated on a 2.5B parameter LLM with comparable performance.

Conclusions:

• LS-PRISM offers an effective and scalable approach for compressing LLMs.
• The method successfully balances model compression with performance preservation.
• LS-PRISM is suitable for deploying LLMs in resource-constrained environments.