为视觉语言模型混合提示学习
Yu Du1,2,3,4,5, Tong Niu1,2,3,4,5, Rong Zhao1,2,3,4,5
1Center for Brain-Inspired Computing Research (CBICR), Tsinghua University, Beijing, China.
Frontiers in artificial intelligence
|June 25, 2025
概括
本研究为视觉语言模型 (VLMs) 引入了一种混合提示的学习方法,该方法提高了适应性和概括性. 这种新的方法使用路由模块来动态选择提示,提高各种下游任务的性能.
科学领域:
- 计算机科学 计算机科学
- 人工智能的人工智能
- 机器学习 机器学习
背景情况:
- 像CLIP这样的视觉语言模型 (VLM) 在下游任务中非常强大.
- 快速学习是VLM的有效适应方法,需要很少的参数.
- 现有的快速学习方法在数据集多样性和过度拟合方面扎.
研究的目的:
- 为了解决VLM单一软快速适应的局限性.
- 提高VLMs的适应性和一般化能力.
- 在快速微调中减轻过问题.
主要方法:
- 提出了一种混合提示的学习方法,用于动态提示选择的路由模块.
- 引入了一个基于硬提示相似性的提示选择门机制.
- 实施语义分组的文本级监督,用于知识保存,具有对比性损失.
主要成果:
- 在11个数据集上显著改进,跨越了少数射击学习,域泛化和基础到新泛化.
- 在调整VLM以适应新任务方面,超越了现有的基准方法.
- 验证了多提示符专业化和知识保存路由的有效性.
结论:
- 拟议的方法有效地弥合了VLM部署中的适应性-通用化权衡.
- 混合提示与路由的学习增强了VLM的性能和稳定性.
- 这种方法提供了一种更有效的方式来适应强大的预先训练有素的VLM.
相关概念视频
Introduction to Learning
551
Learning is the process of acquiring knowledge or skills through practice or experience, leading to long-lasting behavioral changes. This acquisition occurs through interaction with the environment and requires practice or experience. For instance, mastering a skill such as surfing requires considerable practice and experience, highlighting the essential role of repeated interactions with the environment in learning.
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...
551
Observational Learning
321
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
321
Language Development
461
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
461
Vision
55.4K
Vision is the result of light being detected and transduced into neural signals by the retina of the eye. This information is then further analyzed and interpreted by the brain. First, light enters the front of the eye and is focused by the cornea and lens onto the retina—a thin sheet of neural tissue lining the back of the eye. Because of refraction through the convex lens of the eye, images are projected onto the retina upside-down and reversed.
55.4K
Purposive Learning
210
E. C. Tolman emphasized the purposiveness of behavior — the idea that much of our behavior is goal-directed. For instance, employees who aim for a promotion work diligently to meet their targets. Tolman argued that when classical conditioning and operant conditioning occur, the organism acquires certain expectations. In classical conditioning, a child might fear a dog because they expect it to bite. In operant conditioning, a person might consistently work overtime because they expect a...
210
Associative Learning
605
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
605


