Critical Analysis of Deconfounded Pretraining to Improve Visio-Linguistic Models

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Visio-linguistic models often rely on spurious correlations, where a third variable (confounder) influences both image and text.
• AutoDeconfounding was proposed to address this by adjusting for automatically detected confounders.

Purpose of the Study:

• To critically evaluate the effectiveness of AutoDeconfounding in mitigating spurious correlations.
• To investigate the causal link between AutoDeconfounding's deconfounding aspect and performance gains in visio-linguistic tasks.
• To assess the ability of models to identify true confounders and their correlation with task performance.

Main Methods:

• Analysis of the AutoDeconfounding implementation against underlying causal models.
• Ablation studies to isolate the impact of the deconfounding component.
• Creation of a human-labeled causality dataset for empirical validation.
• Evaluation of confounder detection and its relationship to downstream task performance.

Main Results:

• AutoDeconfounding's implementation requires additional assumptions to achieve true deconfounding.
• Performance improvements attributed to AutoDeconfounding were not solely due to its deconfounding aspect.
• Modifying AutoDeconfounding to reduce its deconfounding focus did not harm downstream task performance.
• While some models detect more confounders than random chance, detecting more confounders does not correlate with better visio-linguistic task performance.

Conclusions:

• Current AutoDeconfounding methods have limitations in effectively solving spurious correlations.
• The deconfounding mechanism of AutoDeconfounding is not as effective as previously assumed.
• Future research should focus on developing novel AutoDeconfounding techniques that more reliably address spurious correlations and improve genuine model understanding.