为低资源文本分类进行参数高效微调:对LoRA,IA和ReFT进行比较研究
1Data and Information Science, Faculty of Science and Technology, Rajamangala University of Technology, Pathum Thani, Thailand.
Frontiers in big data
|December 18, 2025
概括
像LoRA和ReFT这样的参数高效微调 (PEFT) 方法在低资源环境中为大型语言模型提供了强大的概括. 在最小可训练参数的情况下,ReFT实现了近乎最佳的性能,性能优于完整的微调.
科学领域:
- 自然语言处理 (NLP) 是一种自然语言处理.
- 机器学习 机器学习
背景情况:
- 大型变压器模型的完整微调在计算上昂贵且数据密集.
- 低资源设置加剧了过度装配和标准微调模型崩等问题.
研究的目的:
- 在统一的低资源条件下实证地比较突出的参数高效微调 (PEFT) 方法.
- 评估PEFT策略的性能和效率之间的权衡.
主要方法:
- 评估低级适应 (LoRA),通过抑制和放大内部激活 (IA3) 输入适应器,以及表示微调 (ReFT).
- 在低资源的AG新闻和亚马逊评论数据集上使用了DistilBERT基础模型.
- 将PEFT方法与精准度,F1分数,可训练参数和GPU内存的完整微调基线进行比较.
主要成果:
- 所有PEFT方法的表现明显超过了完整的微调基线.
- 洛拉获得了最高的F1评分 (亚马逊评论上为0.909).
- ReFT表现出卓越的效率,提供了可比性能 (约98%的LoRA的F1),而培训只有~3%的参数.
结论:
- 在数据稀缺的环境中,PEFT方法对于强大的泛化至关重要,而不仅仅是提高效率.
- ReFT提出了最有效的PEFT策略,平衡高性能与最小的参数训练.
- 本研究提供了一个系统的框架,用于选择基于性能效率权衡的PEFT方法.
相关概念视频
01:31Classification of Systems-II
446
Continuous-time systems have continuous input and output signals, with time measured continuously. These systems are generally defined by differential or algebraic equations. For instance, in an RC circuit, the relationship between input and output voltage is expressed through a differential equation derived from Ohm's law and the capacitor relation,
446
01:26Classification of Systems-I
536
Linearity is a system property characterized by a direct input-output relationship, combining homogeneity and additivity.
Homogeneity dictates that if an input x(t) is multiplied by a constant c, the output y(t) is multiplied by the same constant. Mathematically, this is expressed as:
Homogeneity dictates that if an input x(t) is multiplied by a constant c, the output y(t) is multiplied by the same constant. Mathematically, this is expressed as:
536
01:28Methods of Classification and Identification
936
Bacterial identification relies on a diverse array of techniques to classify and understand microorganisms, each tailored to uncover specific characteristics. Traditional morphological approaches, while still valuable, are limited for closely related or structurally simple organisms. Modern methods integrate biochemical, serological, genetic, and advanced molecular tools to achieve greater accuracy.Morphological and Biochemical TechniquesMorphological characteristics, such as cell shape and...
936
02:07Improving Translational Accuracy
14.0K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
14.0K
02:07Improving Translational Accuracy
3.5K
3.5K
01:22Force Classification
2.2K
Forces play a crucial role in the study of physics and engineering. They are essential in describing the motion, behavior, and equilibrium of objects in the physical world. Forces can be classified based on their origin, type, and direction of action.
Contact and non-contact forces are two of the most widely used categories of forces. As the name suggests, contact forces require physical contact between two objects to act upon each other. Examples of contact forces include frictional,...
Contact and non-contact forces are two of the most widely used categories of forces. As the name suggests, contact forces require physical contact between two objects to act upon each other. Examples of contact forces include frictional,...
2.2K