MFBD : Model-free backdoor defense based on vision-language pre-trained models

Backdoor attacks pose serious security threats to deep neural networks (DNNs), causing networks to output predetermined target labels when specific trigger patterns are embedded in input images. While existing backdoor defense methods utilize additional clean data or victim network models, their effectiveness in defending against various trigger injection methods is limited. We find that vision-language models, such as BLIP-2 and CLIP, are largely unaffected by backdoor triggers in poisoned images. Based on this observation, we propose MFBD, an effective model-free backdoor defense method leveraging the semantic consistency of dual vision-language descriptors. Specifically, we employ BLIP-2 and CLIP to generate descriptions of an image and its saliency masked image, respectively. We then embed both the descriptions and the image labels into a semantic space using SBERT. To distinguish clean and poisoned images, we compute the cosine similarity between their semantic representations. The image exhibiting a low cosine similarity value is classified as a poisoned image. We evaluate a suiute of backdoor defense methods against various attack methods on three datasets. Experimental results demonstrate that our method can effectively mitigate various complex backdoor threats while maintaining high performance across different datasets.