Routing distilled knowledge via mixture of LoRA experts for large language model based bundle generation

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Large Language Models (LLMs) show promise for automatic bundle generation.
• High computational costs limit LLM applicability.
• Knowledge distillation offers efficiency but faces knowledge conflict issues.

Purpose of the Study:

• To develop an efficient framework for bundle generation using distilled knowledge.
• To address knowledge conflicts in distilled LLMs.
• To improve the performance and reliability of bundle generation models.

Main Methods:

• Proposed RouteDK framework using a mixture of Low-Rank Adaptation (LoRA) experts.
• Distilled high-level and fine-grained knowledge from teacher LLMs.
• Implemented a dynamic fusion module with an input-aware router for knowledge integration.
• Introduced an inference-time enhancement module for reliability.

Main Results:

• RouteDK achieved accuracy comparable to or exceeding the teacher LLM.
• Demonstrated strong computational efficiency.
• Outperformed state-of-the-art approaches in bundle generation tasks.
• Effectively mitigated knowledge conflicts through dynamic routing.

Conclusions:

• RouteDK offers an effective and efficient solution for bundle generation.
• The proposed framework successfully integrates diverse distilled knowledge.
• RouteDK enhances LLM performance and reliability for practical applications.