Lightweight Pre-Trained Korean Language Model Based on Knowledge Distillation and Low-Rank Factorization.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computational Linguistics

Background:

• Large Language Models (LLMs) show improved performance with increased size but face deployment challenges on resource-limited devices.
• Pre-trained models for languages other than English, such as Korean, are scarce.
• Efficient NLP solutions are needed for mobile and edge computing environments.

Purpose of the Study:

• To introduce a novel, lightweight pre-trained Korean language model.
• To address the scarcity of Korean NLP resources for constrained devices.
• To demonstrate the effectiveness of knowledge distillation and low-rank factorization for model compression.

Main Methods:

• Knowledge distillation from a large teacher model to smaller student models.
• Low-rank factorization applied to Transformer's feed-forward network and embedding layers.
• Evaluation across six established Korean Natural Language Processing tasks.

Main Results:

• The most compact model (KR-ELECTRA-Small-KD) achieved over 97.387% of the teacher model's performance with an 8.15× size reduction.
• KR-ELECTRA-Small-KD surpassed the teacher model on the NSMC sentiment classification benchmark with 89.720% accuracy.
• Demonstrated significant parameter reduction while maintaining high performance.

Conclusions:

• The proposed lightweight Korean language model is effective for resource-constrained NLP applications.
• Knowledge distillation and low-rank factorization are viable techniques for creating efficient language models.
• This work facilitates the deployment of advanced NLP capabilities on mobile and edge devices for Korean language processing.