VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Large Vision-Language Models (LVLMs) advance artificial intelligence but raise concerns about inherent biases.
• Existing benchmarks inadequately assess LVLM biases due to limited data and scope.

Purpose of the Study:

• To introduce VLBiasBench, a comprehensive benchmark for evaluating social biases in LVLMs.
• To address the limitations of current benchmarks in scale, question format, and bias categories.

Main Methods:

• Generated 46,848 images using Stable Diffusion XL, creating 128,342 samples with diverse questions.
• Dataset covers nine social bias categories (e.g., race, gender) and two intersectional categories.
• Utilized both open-ended and close-ended questions for multifaceted bias evaluation.

Main Results:

• Conducted extensive evaluations on 17 LVLMs (15 open-source, 2 closed-source).
• Uncovered and provided new insights into the specific biases present in these advanced models.
• Demonstrated the effectiveness of VLBiasBench in identifying model biases.

Conclusions:

• VLBiasBench offers a robust framework for assessing and mitigating biases in LVLMs.
• Highlights the critical need for comprehensive bias evaluation in the development of general artificial intelligence.
• The benchmark and findings contribute to more equitable AI development.