持续的预训练可以减轻语言和视觉方面的遗忘
Andrea Cossu1, Antonio Carta1, Lucia Passaro1
1University of Pisa, Largo B. Pontecorvo, 3, Pisa, 56127, Italy.
概括
持续的预训练,在任务微调之前,模型从流数据中学习,可以减轻忘记. 在没有额外的持续学习策略的情况下,自然语言处理 (NLP) 和视觉方面的自我监督预培训被证明是有效的.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 深度学习 (Deep Learning) 是一种深度学习.
背景情况:
- 预训练的模型通常初始化持续学习 (CL),但很少在CL*期间进行预训练.
- 调查持续预培训 (CPT),模型在下游任务微调之前适应数据流至关重要.
研究的目的:
- 评估持续预训练场景及其对模型遗忘的影响.
- 分析影响CPT忘记的因素,包括输入模式,架构和预训练协议.
- 引入样本效率预培训 (SEP) 方法,以加快预培训阶段.
主要方法:
- 使用忘记控制数据集开发了CPT的评估协议.
- 解开了输入模式 (NLP,视觉),架构 (变压器,ResNet) 和预训练 (监督,自我监督) 对遗忘的影响.
- 提出并评估了样本效率预训练 (SEP) 方法.
主要成果:
- 训练前的程序显著影响忘记,比其他因素更重要.
- 在NLP和视觉中,自主监督的CPT有效地减轻了没有专门的CL技术的忘记.
- 模型深度,输入模式和架构类型等因素在减少遗忘方面不那么重要.
结论:
- 自主监督的持续预训是一个强大的,令人惊的简单策略,以防止灾难性的遗忘.
- 对于非静止数据流来说,CPT,特别是自主监督,提供了一个强大的方法来构建可适应的模型.
相关概念视频
Long-term Potentiation
55.1K
Long-term potentiation, or LTP, is one of the ways by which synaptic plasticity—changes in the strength of chemical synapses—can occur in the brain. LTP is the process of synaptic strengthening that occurs over time between pre- and postsynaptic neuronal connections. The synaptic strengthening of LTP works in opposition to the synaptic weakening of long-term depression (LTD) and together are the main mechanisms that underlie learning and memory.
55.1K
Language and Cognition
340
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
340
Forgetting
67
Forgetting is an intrinsic aspect of human memory, characterized by the gradual loss or inaccessibility of information over time. Hermann Ebbinghaus, a pioneering psychologist, extensively studied this phenomenon and formulated the forgetting curve. This curve illustrates that memory loss occurs rapidly immediately after learning and then decelerates over time. Several mechanisms contribute to forgetting, including encoding failure, storage decay, retrieval failure, and interference.
Encoding...
Encoding...
67
Role of Cerebellum and Prefrontal Cortex in Memory
413
The cerebellum, while traditionally associated with motor control, also plays a crucial role in memory, particularly in procedural memory, which involves learning motor tasks that become automatic through repetition. For example, studies have shown that when the cerebellum is damaged, individuals or animals lose the ability to learn conditioned motor responses, such as the conditioned eye-blink response in classical conditioning experiments with rabbits. This study demonstrates the...
413
Interference and Decay
127
Forgetting is a complex cognitive phenomenon influenced by several factors, among which interference and decay are particularly prominent. These processes explain why individuals often struggle to retrieve specific information from memory, leading to lapses in recall that can be observed in everyday situations.
Interference occurs when competing memories hinder the retrieval of particular information. It can be classified into two types: proactive and retroactive interference. Proactive...
Interference occurs when competing memories hinder the retrieval of particular information. It can be classified into two types: proactive and retroactive interference. Proactive...
127
Implicit Memories
119
Implicit memories, also known as non-declarative memories, are long-term memories that function outside of conscious awareness. These memories influence behavior and skills without explicit knowledge. This type of memory is evident in tasks like playing tennis, snowboarding, and texting. Implicit memory has three subsystems: procedural memory, conditioning, and priming. This type of memory is essential in various activities, from everyday tasks to specialized skills.
One key aspect of implicit...
One key aspect of implicit...
119


