梯度投影用于连续参数-高效调
概括
参数效率调整 (PETs) 现在可以使用直角梯度投影抵抗遗忘旧知识. 这种新的参数高效梯度预测 (PEGP) 框架可以改善大型模型的持续学习.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 深度学习 (Deep Learning) 是一种深度学习.
背景情况:
- 参数效率调整 (PET) 为大型模型提供了高效的训练,但在学习新信息和保留旧知识之间的权衡方面扎,导致泛化崩和跨模式幻觉.
- 现有的PET方法如Adapter,LoRA,Prefix-tuning和Prompt-tuning在持续学习场景中面临着挑战.
研究的目的:
- 提出一个统一的框架,参数高效梯度预测 (PEGP),解决PET的知识遗忘问题.
- 从理论和经验上证明直角梯度投影在减轻大规模模型内遗忘的有效性.
主要方法:
- 从梯度投影的角度重新制定现有的PET (适配器,LoRA,前调整,快速调整).
- 在各种PET范式中引入直角梯度投影.
- 从理论上分析直角梯度投影如何通过最小化对旧特征空间的影响来抵抗遗忘.
主要成果:
- 拟议的PEGP框架有效地减少了各种持续学习环境中的遗忘,包括类增量,领域增量,任务增量和多模式的持续学习.
- 对ViT和CLIP骨干的实验证明了PEGP在减少遗忘方面的效率,而不会显著增加记忆或训练时间.
- 在理论上,正角梯度投影被证明是有效的,即使在大型模型中,也可以抵御遗忘.
结论:
- PEGP提供了一种统一而有效的方法,以减轻大型模型的参数效率调整方法中的灾难性遗忘.
- 正交梯度投影机制提供了一种原则性的方法,以平衡学习新知识与保存现有知识.
- 这个框架在各种持续学习任务和模型架构中具有广泛的适用性.
更多相关视频
11:26Assessing Cerebral Autoregulation via Oscillatory Lower Body Negative Pressure and Projection Pursuit Regression
Published on: December 10, 2014
12.5K
15:18Near Infrared Optical Projection Tomography for Assessments of β-cell Mass Distribution in Diabetes Research
Published on: January 12, 2013
16.5K
相关概念视频
Fischer Projections
13.8K
Learning to draw Fischer projections of molecules and understanding their relevance plays a crucial role in the visual depiction of organic molecules. A Fischer projection is a two-dimensional projection on a planar surface to simplify the three-dimensional wedge–dash representation of molecules. This is especially helpful in the case of molecules with multiple chiral centers that can be difficult to draw. Here, all the bonds of interest are represented as horizontal or vertical lines.
13.8K
Calibration Curves: Linear Least Squares
2.1K
A calibration curve is a plot of the instrument's response against a series of known concentrations of a substance. This curve is used to set the instrument response levels, using the substance and its concentrations as standards. Alternatively, or additionally, an equation is fitted to the calibration curve plot and subsequently used to calculate the unknown concentrations of other samples reliably.
For data that follow a straight line, the standard method for fitting is the linear...
For data that follow a straight line, the standard method for fitting is the linear...
2.1K
Coordinates and Map Projections
150
Coordinates and map projections are essential tools in accurately representing the Earth's surface for various applications, ranging from navigation to spatial analysis. The latitude and longitude coordinate system is a universally recognized framework for defining locations. Latitude specifies the distance of a point north or south of the equator, measured in degrees from 0° at the equator to 90° at the poles. Longitude indicates a location's position east or west of the prime meridian,...
150
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
100
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
100
Linear Approximation in Frequency Domain
131
Linear systems are characterized by two main properties: superposition and homogeneity. Superposition allows the response to multiple inputs to be the sum of the responses to each individual input. Homogeneity ensures that scaling an input by a scalar results in the response being scaled by the same scalar.
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear....
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear....
131
PD Controller: Design
345
In automotive engineering, car suspension systems often employ Proportional Derivative (PD) controllers to enhance performance. PD controllers are utilized to adjust the damping force in response to road conditions. A controller, acting as an amplifier with a constant gain, demonstrates proportional control, with output directly mirroring input.
Designing a continuous-data controller requires selecting and linking components like adders and integrators, which are fundamental in Proportional,...
Designing a continuous-data controller requires selecting and linking components like adders and integrators, which are fundamental in Proportional,...
345
