FM-LoRA:为持续学习提供因子化的低等级元诱导
Xiaobing Yu1, Jin Yang1, Xiao Wu1,2
1Dept. of Radiology, Washington University in St. Louis, St. Louis, USA.
概括
本研究介绍了FM-LoRA,这是一种高效的持续学习方法,可以避免顺序任务的参数增长. 它增强了在各种任务和领域的模型适应和知识保留.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 计算机视觉 计算机视觉
背景情况:
- 持续学习 (CL) 旨在使预先训练的模型适应连续的任务,而不会发生灾难性的遗忘.
- 现有的CL方法经常受到参数扩展的影响,缺乏任务相似性意识,阻碍了性能.
- 变压器是强大的预训练模型,但需要有效的适应策略来进行顺序学习.
研究的目的:
- 提出FM-LoRA,一种新且高效的低级适应方法,用于持续学习.
- 为了解决参数增长和提高任务相似性意识,在持续适应中.
- 为了使学习能够在各种顺序的任务和领域中学习可概括的模型.
主要方法:
- FM-LoRA集成了一个动态排名选择器 (DRS) 和动态元提示 (DMP).
- 它利用一个共享的低级子空间来保护知识并有效地分配模型容量.
- 该方法在使用变压器的类增量学习 (CIL) 和域增量学习 (DIL) 基准上进行了评估.
主要成果:
- FM-LoRA在持续学习中有效缓解了灾难性遗忘.
- 该方法在各种任务和领域实现了强大的性能,包括ImageNet-R,CIFAR100,CUB200和DomainNet.
- FM-LoRA避免了持续的参数扩展,为顺序适应提供了有效的解决方案.
结论:
- FM-LoRA提供了一种高效有效的方法,用于连续任务的持续学习.
- 拟议的方法增强了不同类别和领域的模型通用性.
- FM-LoRA提供了一种可持续的解决方案,用于在动态学习环境中适应大型预训练模型.
相关概念视频
Purposive Learning
442
E. C. Tolman emphasized the purposiveness of behavior — the idea that much of our behavior is goal-directed. For instance, employees who aim for a promotion work diligently to meet their targets. Tolman argued that when classical conditioning and operant conditioning occur, the organism acquires certain expectations. In classical conditioning, a child might fear a dog because they expect it to bite. In operant conditioning, a person might consistently work overtime because they expect a...
442
Observational Learning
832
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
832
Cognitive Learning
1.0K
Cognitive learning is based on purposive behavior, incidental learning, and insight learning.
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
1.0K
Long-term Potentiation
58.3K
Long-term potentiation, or LTP, is one of the ways by which synaptic plasticity—changes in the strength of chemical synapses—can occur in the brain. LTP is the process of synaptic strengthening that occurs over time between pre- and postsynaptic neuronal connections. The synaptic strengthening of LTP works in opposition to the synaptic weakening of long-term depression (LTD) and together are the main mechanisms that underlie learning and memory.
58.3K
Long-term Potentiation
3.4K
Long-term potentiation, or LTP, is one of the ways by which synaptic plasticity—changes in the strength of chemical synapses—can occur in the brain. LTP is the process of synaptic strengthening that occurs over time between pre and postsynaptic neuronal connections. The synaptic strengthening of LTP works in opposition to the synaptic weakening of long-term depression (LTD) and together are the main mechanisms that underlie learning and memory.
Hebbian LTP
LTP can occur when...
Hebbian LTP
LTP can occur when...
3.4K
Metacognition
709
Metacognition is a conscious process where individuals are aware of their cognitive and executive processes, such as planning before solving a problem or self-monitoring during reading. For instance, a writer may need help with composing a piece. The situation involves a writer who is working on a piece of writing, but while doing so, they realize that something is missing. They notice that their characters lack depth or details. This realization occurs because the writer is reflecting on their...
709

