合成-CLIP:合成数据使CLIP在数据有限的场景中更好地泛化
Mushui Liu1, Weijie He1, Ziqian Lu2
1College of Information Science and Electronic Engineering, Zhejiang University, China.
概括
Synth-CLIP使用合成数据增强视觉语言模型 (VLMs),以改善对新课程的概括,即使现实数据有限. 这种方法有效地扩大了数据集,并丰富了类别的多样性,以提高性能.
科学领域:
- 计算机视觉 计算机视觉
- 机器学习 机器学习
- 人工智能的人工智能
背景情况:
- 快速学习有效地将视觉语言模型 (VLM) 转移到下游任务中.
- 仅在基础类上微调VLMs会阻碍对新型类的概括,尤其是在有限的数据的情况下.
研究的目的:
- 提出Synth-CLIP,一种利用合成数据的创新方法,以提高VLM对基础类和新类的概括性.
- 提高像CLIP这样的预训练模型的适应新类别的能力,而无需视觉样本.
主要方法:
- Synth-CLIP微调预训练的CLIP模型,并为视觉样本提供量身定制的,域特定的和共享的提示.
- 集成真实和合成数据,将视觉特征重新组织到语义空间中,以扩大数据池和多样性.
- 引入基于语义一致性的跨域特征对齐损失,以匹配基于语义一致性的特征嵌入空间中的真实和合成样本.
主要成果:
- Synth-CLIP有效地扩大了数据库,丰富了类别的多样性.
- 将视觉和语义分布与合成数据对齐有助于重新平衡决策边界,即使没有新的视觉样本.
- 实验结果显示了跨基准的竞争性表现,在开放词汇场景中,在11个数据集中的新类别中,PromptSRC的表现超过了3.0%.
结论:
- Synth-CLIP提供了一种强大的方法来改进VLM概括,特别是在新类的低数据场景中.
- 使用合成数据和跨领域对齐对于提高模型适应性和性能至关重要.
- 这种方法表明了对开放词汇识别和少数射击学习任务的巨大潜力.
更多相关视频
相关概念视频
Survival Tree
57
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a...
Building a Survival Tree
Constructing a...
57
Generalization, Discrimination, and Extinction
428
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
428
Improving Translational Accuracy
8.7K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
8.7K
Case Studies
11.6K
There are many research methods available to psychologists in their efforts to understand, describe, and explain behavior and the cognitive and biological processes that underlie it.
11.6K
Naturalistic Observations
15.4K
If you want to understand how behavior occurs, one of the best ways to gain information is to simply observe the behavior in its natural context. However, people might change their behavior in unexpected ways if they know they are being observed. How do researchers obtain accurate information when people tend to hide their natural behavior? As an example, imagine that your professor asks everyone in your class to raise their hand if they always wash their hands after using the restroom. Chances...
15.4K
Censoring Survival Data
62
Survival analysis is a statistical method used to analyze time-to-event data, often employed in fields such as medicine, engineering, and social sciences. One of the key challenges in survival analysis is dealing with incomplete data, a phenomenon known as "censoring." Censoring occurs when the event of interest (such as death, relapse, or system failure) has not occurred for some individuals by the end of the study period or is otherwise unobservable, and it might have many different...
62


