LADDER: Language-Driven Slice Discovery and Error Rectification in Vision Classifiers

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Current slice discovery methods for pre-trained vision models rely on predefined attributes, limiting bias detection.
• Existing approaches fail to incorporate common sense or domain-specific knowledge and overlook preprocessing-induced biases.
• Bias-inducing variables in AI models often leave traces in unstructured text data like logs.

Purpose of the Study:

• To introduce LADDER, a novel method for identifying systematic biases in pre-trained vision models.
• To leverage Large Language Models (LLMs) for bias hypothesis generation and mitigation.
• To address limitations of attribute-based methods by utilizing unstructured text data and LLM reasoning.

Main Methods:

• LADDER projects internal model activations into text using a retrieval approach, prompting LLMs for bias hypotheses.
• It converts preprocessing data into text to detect biases introduced during data preparation.
• The method generates pseudo-labels for identified biases, enabling mitigation without manual attribute annotations.

Main Results:

• LADDER successfully identifies biases by leveraging LLM reasoning and domain knowledge from text data.
• The approach effectively detects biases originating from both image attributes and preprocessing pipelines.
• Evaluations across natural and medical imaging datasets demonstrate LADDER's consistent outperformance over existing methods.

Conclusions:

• LADDER offers a robust and versatile solution for bias discovery and mitigation in pre-trained vision models.
• Utilizing LLMs with unstructured text data represents a significant advancement in AI fairness and reliability.
• The method reduces the need for expensive manual annotations, making bias mitigation more accessible.