VisMoDAI: Visual Analytics for Evaluating and Improving Corruption Robustness of Vision-Language Models.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Vision-language (VL) models excel at multi-modal comprehension but struggle with real-world data corruption and distribution shifts.
• Existing methods for evaluating VL model robustness lack in-depth understanding of model behavior and require significant expertise.
• Data augmentation (DA) is crucial for improving robustness, but effective strategy formulation is challenging.

Purpose of the Study:

• To introduce VisMoDAl, a visual analytics framework for evaluating VL model robustness against various data corruption types.
• To identify underperformed samples and guide the development of effective data augmentation strategies.
• To facilitate a deeper understanding of how data corruption impacts VL model behavior.

Main Methods:

• VisMoDAl supports multi-level analysis, from specific corruption performance to task-driven inspection of model behavior.
• The framework enables users to reason about the effects of corruption on VL models.
• Case studies and quantitative evaluations on image captioning task demonstrate the system's utility.

Main Results:

• VisMoDAl provides a visual analytics approach to understand VL model behavior under data corruption.
• The framework aids in identifying specific weaknesses and underperformed data samples.
• It facilitates the formulation of targeted data augmentation strategies.

Conclusions:

• VisMoDAl enhances the evaluation of VL model robustness against data corruption.
• The framework promotes better understanding of model behavior and guides effective data augmentation.
• This visual analytics approach is valuable for developing more resilient VL models for practical applications.