MDAPT:多模态深度对抗快速调整,以提高视觉语言模型的对抗稳定性
Chao Li1, Yonghao Liao1, Caichang Ding2
1School of Computer Science, Hubei University of Technology, Wuhan 430068, China.
Sensors (Basel, Switzerland)
|January 11, 2025
概括
本研究介绍了多模式深度对抗即时调整 (MDAPT),以增强视觉语言模型 (VLM) 对抗对抗攻击. MDAPT显著提高了VLM的准确性和稳定性,超过了传统方法.
科学领域:
- 人工智能的人工智能
- 计算机视觉 计算机视觉
- 自然语言处理自然语言处理.
背景情况:
- 像CLIP这样的大型视觉语言模型 (VLM) 在性能方面表现出色,但易受对抗性示例的影响.
- 现有的方法往往难以保持对复杂攻击的稳定性.
研究的目的:
- 从多模式的角度研究VLM的准确性和稳定性.
- 提出一种新的微调方法,以提高VLM对抗对方攻击的弹性.
主要方法:
- 开发了一种微调技术,即多模式深度对抗快速调整 (MDAPT).
- MDAPT引导使用文本提示生成视觉提示,以提高VLM性能.
- 在对抗条件下对三个数据集进行了广泛的实验.
主要成果:
- 在三个数据集 (ε=4/255) 上实现了显著的性能改进.
- 与手动提示相比,MDAPT的准确性平均提高了17.84%,稳定性增加了10.85%.
- 在具有高效设置的三种不同的攻击方法下,证明了实质性的收益 (32.16%的准确性,21.00%的稳定性).
结论:
- MDAPT有效地提高了VLMs的准确性和稳定性,以应对对立的例子.
- 拟议的多模式方法为开发更具弹性的人工智能系统提供了一个有希望的方向.
- 在对抗性环境中,MDAPT比传统的提示设计策略提供了显著的优势.
相关概念视频
Improving Translational Accuracy
8.7K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
8.7K
Masking and Demasking Agents
2.3K
EDTA titrations may necessitate masking and demasking agents to temporarily protect a particular metal ion in a mixture from the EDTA reaction. These agents facilitate the sequential analysis of the metal ions by forming stable complexes with some—but not all—metal ions during certain steps.
There are many masking agents, such as cyanide, fluoride, triethanolamine, thiourea, and 2,3-bis(sulfanyl)propan-1-ol (formerly 2,3-dimercapto-1-propanol), with the masking agent chosen based on...
There are many masking agents, such as cyanide, fluoride, triethanolamine, thiourea, and 2,3-bis(sulfanyl)propan-1-ol (formerly 2,3-dimercapto-1-propanol), with the masking agent chosen based on...
2.3K
Multi-input and Multi-variable systems
96
Cruise control systems in cars are designed as multi-input systems to maintain a driver's desired speed while compensating for external disturbances such as changes in terrain. The block diagram for a cruise control system typically includes two main inputs: the desired speed set by the driver and any external disturbances, such as the incline of the road. By adjusting the engine throttle, the system maintains the vehicle's speed as close to the desired value as possible.
In the absence...
In the absence...
96
Depth Perception and Spatial Vision
548
Depth perception is the ability to perceive objects three-dimensionally. It relies on two types of cues: binocular and monocular. Binocular cues depend on the combination of images from both eyes and how the eyes work together. Since the eyes are in slightly different positions, each eye captures a slightly different image. This disparity between images, known as binocular disparity, helps the brain interpret depth. When the brain compares these images, it determines the distance to an object.
548
Associative Learning
287
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
287
Language Development
317
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
317
