通过语义意识微调来增强少数镜头的CLIP
IEEE transactions on neural networks and learning systems
|August 26, 2024
概括
微调的CLIP的注意力聚合层通过适应特定任务的语义来增强少数镜头的学习. 这种有语义意识的微调提高了低资源任务的性能,优于现有的方法.
科学领域:
- 计算机科学 计算机科学
- 人工智能的人工智能
- 机器学习 机器学习
背景情况:
- 深度神经网络在低资源场景中从有限的数据中学习一般化表示.
- 对比性语言图像预训练 (CLIP) 在少数镜头的适应方面表现有前途,但结其参数可能会阻碍下游任务的性能.
- 现有的方法经常结CLIP参数,以防止过拟合和灾难性遗忘,可能忽视任务特定的语义需求.
研究的目的:
- 通过调整CLIP的视觉编码器以适应下游任务来提高少数镜头学习性能.
- 为了解决CLIP中固定注意力聚合的局限性,用于各种短暂的场景.
- 开发一种微调任务特定语义的方法,同时保留CLIP的一般知识.
主要方法:
- 建议微调CLIP视觉编码器的注意力聚合层,以专注于特定任务的语义.
- 在推断过程中引入了剩余混合,以结合微调和原始的CLIP特征.
- 开发了语义意识微调 (SAF) 并将其与适配器方法 (称为SAF-Adapter) 集成.
主要成果:
- 语义意识微调 (SAF) 显著提高了11个基准点的CLIP性能.
- 在一拍 (1.51-2.38%) 和四拍 (0.48-1.37%) 设置中,SAF和SAF-Adapter的性能远远超过了第二好的方法.
- 拟议的方法在资源较少的,短暂的适应任务中表现出有效性.
结论:
- 微调CLIP的注意力聚合层是改善少量学习的可行策略.
- 语义意识微调有效地将模型适应特定任务的细微差别,促进概括.
- 拟议的方法在深度学习中为少数镜头的适应提供了显著的改进.
更多相关视频
相关概念视频
Improving Translational Accuracy
9.5K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
9.5K
Extraction: Advanced Methods
433
Metal ions can be separated from one another by complexation with organic ligands–the chelating agent– to form uncharged chelates. Here, the chelating agent must contain hydrophobic groups and behave as a weak acid, losing a proton to bind with the metal. Since most organic ligands used in this process are insoluble or undergo oxidation in the aqueous phase, the chelating agent is initially added to the organic phase and extracted into the aqueous phase. The metal-ligand complex is...
433
Chunking and Rehearsal in Sensory Memory
186
Improving short-term memory can be achieved through techniques like chunking and rehearsal. Chunking involves organizing information into larger, more manageable units. This technique is particularly useful for information that exceeds the typical memory span of between five and nine items. For instance, logging into an online account with a password like "ta89vq0179gz" involves grouping letters and numbers into three chunks—ta89, vq01, and 79gz. It makes large amounts of...
186
Force Classification
1.2K
Forces play a crucial role in the study of physics and engineering. They are essential in describing the motion, behavior, and equilibrium of objects in the physical world. Forces can be classified based on their origin, type, and direction of action.
Contact and non-contact forces are two of the most widely used categories of forces. As the name suggests, contact forces require physical contact between two objects to act upon each other. Examples of contact forces include frictional,...
Contact and non-contact forces are two of the most widely used categories of forces. As the name suggests, contact forces require physical contact between two objects to act upon each other. Examples of contact forces include frictional,...
1.2K
Reducing Line Loss
150
In a three-phase circuit, line loss is an indicator of energy dissipated as heat due to the resistance of transmission lines. To address this, incorporating transformers into the system—a step-up transformer at the source and a step-down transformer at the load—is a strategic solution. Two three-phase transformers are introduced to improve this.
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss...
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss...
150
Downsampling
141
When considering a sampled sequence with zero values between sampling instants, one can replace it by taking every N-th value of the sequence. At these integer multiples of N, the original and sampled sequences coincide. This process, known as decimation, involves extracting every N-th sample from a sequence, thereby creating a more efficient sequence.
The Fourier transform of the decimated sequence reveals a combination of scaled and shifted versions of the original spectrum. This...
The Fourier transform of the decimated sequence reveals a combination of scaled and shifted versions of the original spectrum. This...
141


