3VL: Using Trees to Improve Vision-Language Models' Interpretability.

Area of Science:

• Computer Science
• Artificial Intelligence
• Natural Language Processing

Background:

• Vision-Language models (VLMs) excel at aligning image and text but struggle with compositional language concepts (CLC).
• Current VLMs lack interpretability, hindering debugging and mitigation of failures in understanding attributes, states, and relations.
• Compositional reasoning is crucial for advanced visual understanding tasks.

Purpose of the Study:

• To introduce the Tree-augmented Vision-Language (3VL) model architecture and training technique.
• To enhance the compositional reasoning capabilities of VLMs.
• To improve the interpretability of VLMs for debugging and understanding failures.

Main Methods:

• Expanding image-text pairs into hierarchical tree structures using language analysis.
• Inducing the hierarchical text structure into the model's visual representation.
• Utilizing the Anchor inference method for text unification and filtering nuisance factors.
• Employing the Differential Relevance (DiRe) tool for model interpretability via relevancy map comparison.

Main Results:

• The 3VL model demonstrates enhanced interpretability and compositional reasoning.
• The Anchor method effectively filters nuisance factors, improving CLC understanding performance on benchmarks like VL-Checklist.
• DiRe provides compelling visualizations explaining model successes and failures.

Conclusions:

• The 3VL model, coupled with Anchor and DiRe, offers a significant advancement in VLM capabilities for compositional language understanding.
• Improved interpretability facilitates the debugging and refinement of VLMs.
• This work addresses critical limitations in current VLMs, paving the way for more robust and understandable AI systems.