通过学习注入知识来调整视觉语言模型
概括
本研究引入了一种用于视觉语言模型 (VLMs) 的新知识注入框架. 它通过注入任务无关的知识特征来增强对新任务的概括性,改善各种设置中的性能.
科学领域:
- 计算机视觉 计算机视觉
- 机器学习 机器学习
- 人工智能的人工智能
背景情况:
- 预先训练的视觉语言模型 (VLMs),如CLIP,由于广泛的图像文本训练,在零拍摄任务中表现出色.
- 目前的方法通常依赖于特定任务的提示,限制VLM适应未见任务的适应性.
- 在他们的文本编码器中,VLM 拥有大量的记忆外观知识.
研究的目的:
- 开发一个可推广的适应框架,使VLM能够下游视觉任务.
- 通过利用无关任务的知识来克服任务特定提示的局限性.
- 为了提高VLM的性能,少量学习,概括和域移动.
主要方法:
- 提出了一个知识注入框架,使用任务无意识的知识特征.
- 通过VLM的文本编码器从可学习的提示句中提取多层特征.
- 引入了一个知识注入模块 (KIM) 以提取知识特征来完善图像或文本特征.
主要成果:
- 拟议的框架显著提高了对类别内差异的区分能力和稳定性.
- 与最近的方法相比,在少数射击学习,基础到新类的概括,跨数据集转移和域概括方面取得了更好的表现.
- 在少量学习中,CoCoOp的表现超过了4.5%,在基础到新类的概括中,CoCoOp的表现超过了4.4%.
结论:
- 知识注入框架使两种模式都能够从VLM的文本编码器知识中受益.
- 展示了VLMs对各种下游视觉任务的有效和可通用的适应.
- 该方法为改善VLM性能和适应性提供了一个有希望的方向.
相关概念视频
Improving Translational Accuracy
9.3K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
9.3K
Associative Learning
308
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
308
Vision
53.0K
Vision is the result of light being detected and transduced into neural signals by the retina of the eye. This information is then further analyzed and interpreted by the brain. First, light enters the front of the eye and is focused by the cornea and lens onto the retina—a thin sheet of neural tissue lining the back of the eye. Because of refraction through the convex lens of the eye, images are projected onto the retina upside-down and reversed.
53.0K
Model Approaches for Pharmacokinetic Data: Distributed Parameter Models
63
Pharmacokinetic models are mathematical constructs that represent and predict the time course of drug concentrations in the body, providing meaningful pharmacokinetic parameters. These models are categorized into compartment, physiological, and distributed parameter models.
The distributed parameter models are specifically designed to account for variations and differences in some drug classes. This model is particularly useful for assessing regional concentrations of anticancer or...
The distributed parameter models are specifically designed to account for variations and differences in some drug classes. This model is particularly useful for assessing regional concentrations of anticancer or...
63
Information Processing Approach
31
The information-processing theory of cognitive development centers on fundamental mental processes, including attention, memory, and problem-solving skills. Researchers in this field examine how cognitive abilities, such as working memory, evolve and influence children's overall development. Studies indicate that children with stronger working memory tend to excel in reading comprehension, math, and problem-solving compared to peers with less efficient memory skills. Low working memory is...
31
Generalization, Discrimination, and Extinction
459
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
459


