Toward a Holistic Evaluation of Robustness in CLIP Models

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Contrastive Language-Image Pre-training (CLIP) models excel in zero-shot classification but require deeper robustness analysis.
• Existing evaluations focus on overall accuracy, neglecting specific visual factors, safety objectives, and cross-modal understanding.

Purpose of the Study:

• To comprehensively assess CLIP model robustness across visual factors, safety metrics (uncertainty, OOD detection), and cross-modal finesse.
• To extend evaluation to 3D awareness and analyze interactions within large multimodal models (LMMs) using CLIP.
• To investigate the impact of model architecture, training, fine-tuning, and prompting on CLIP's performance.

Main Methods:

• Evaluated CLIP robustness against variations in visual factors and assessed confidence uncertainty and out-of-distribution detection.
• Examined 3D awareness and the interplay between vision/language encoders in LMMs leveraging CLIP.
• Analyzed the influence of six factors: architecture, training distribution/set size, fine-tuning, loss, and prompts.

Main Results:

• CLIP architecture significantly impacts robustness to 3D corruption; models show a shape bias, which decreases post-fine-tuning.
• LLaVA models, using CLIP encoders, outperform CLIP alone on challenging categories.
• Specific factors like architecture and fine-tuning critically influence CLIP's robustness and safety.

Conclusions:

• CLIP models exhibit nuanced robustness and safety characteristics influenced by various factors.
• Further research into 3D awareness and LMM interactions is crucial for advancing reliable AI.
• Findings provide guidance for developing more robust and dependable CLIP-based vision-language systems.