Toward Efficient and Generalizable Text Dataset Distillation via a Dual-Agent Large Language Model Framework.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Large-scale datasets incur high training costs for machine learning models.
• Text dataset distillation is challenging due to text's discrete nature and limitations of existing methods.
• Existing gradient-matching and embedding optimization methods for text distillation are often ineffective or inefficient.

Purpose of the Study:

• To propose an efficient, automated, and general-purpose text dataset distillation framework.
• To address the challenges of discrete text data and improve distillation performance.
• To develop an LLM-native approach using dual-agent collaboration for text dataset distillation.

Main Methods:

• A dual-agent collaboration framework with 'Selector' and 'Improver' agents.
• Multi-dimensional scoring for sample selection and enhancement under semantic consistency.
• Automated self-iterative cycle of generation and selection driven by scoring signals for agent self-reward.

Main Results:

• Achieved full-dataset performance with only 2.5% of original data in three iterations for Llama-2-7B, Mistral-7B, and Qwen2.5-7B on MMLU and Winogrande.
• Demonstrated improved training stability, convergence efficiency, and cross-model generalization.
• Maintained a competitive distillation cost of 143 GPU hours.

Conclusions:

• The proposed LLM-native distillation framework offers a practical solution for efficient and automated text dataset distillation.
• The dual-agent approach effectively overcomes limitations of traditional methods for discrete text data.
• The distilled datasets enable significant performance gains and training efficiencies for large language models.