Interpreting vision and language generative models with semantic visual priors.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Explainability methods for image-to-text models face challenges with token-by-token explanations, which are computationally expensive and lack comprehensive insight.
• Current methods often use superpixels as features, hindering interpretability due to their lack of semantic meaning in visual explanations.

Purpose of the Study:

• To develop a novel framework for generating comprehensive and semantically meaningful explanations for image-to-text models.
• To improve the efficiency and interpretability of visual explanations in large-scale vision-language models.

Main Methods:

• Developed a framework based on SHAP (SHapley Additive exPlanations) to leverage the meaning representation of the entire output sequence.
• Exploited semantic priors in the visual backbone to extract an arbitrary number of semantically meaningful features.
• Enabled efficient computation of Shapley values for large-scale models, generating highly interpretable visual explanations.

Main Results:

• The proposed method generates semantically more expressive explanations compared to traditional approaches.
• Achieved lower computational cost for generating explanations.
• Demonstrated generalization capabilities across a wide range of vision-language models.

Conclusions:

• The developed framework offers a significant advancement in explainability for image-to-text models.
• Provides a more efficient and interpretable approach to understanding model behavior.
• The method is adaptable and broadly applicable to various vision-language architectures.