LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Large Vision-Language Models (LVLMs) are pivotal in multimodal AI, yet lack standardized holistic evaluations.
• Existing assessments often fail to capture the full spectrum of LVLM capabilities in diverse scenarios.

Purpose of the Study:

• To establish a comprehensive evaluation framework, the LVLM evaluation Hub (LVLM-eHub), for publicly available LVLMs.
• To quantitatively and qualitatively assess the efficacy of leading LVLMs in multimodal understanding tasks.
• To investigate the impact of model configurations, alignment mechanisms, and training data on LVLM performance.

Main Methods:

• Developed the LVLM evaluation Hub (LVLM-eHub) encompassing 13 representative LVLMs.
• Conducted quantitative capability evaluations across five categories (e.g., VQA, object hallucination) using 42 benchmarks.
• Implemented an online arena platform for user-level, open-world question-answering evaluations.

Main Results:

• Identified key factors influencing LVLM performance, including model architecture and training data composition.
• Demonstrated significant performance variations among different LVLMs in both quantitative and user-based assessments.
• Uncovered innovative findings regarding the strengths and weaknesses of current LVLM approaches.

Conclusions:

• The LVLM-eHub provides a robust framework for the systematic evaluation and development of advanced multimodal AI.
• Findings offer critical insights for researchers and developers aiming to enhance LVLM capabilities.
• The study lays the groundwork for future innovations in multimodal learning strategies and evaluation methodologies.