VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Advancements in video generation models necessitate robust evaluation methods.
• Current metrics for video generation quality often fail to align with human perception.
• A comprehensive benchmark is crucial for guiding future developments in AI video synthesis.

Purpose of the Study:

• To introduce VBench++, a comprehensive and hierarchical benchmark suite for evaluating video generation models.
• To dissect video generation quality into 16 specific, disentangled dimensions with tailored evaluation methods.
• To assess both technical quality and trustworthiness of generative video models.

Main Methods:

• VBench++ comprises 16 dimensions for evaluating text-to-video and image-to-video generation.
• Human preference annotations were collected to validate benchmark alignment with human perception.
• An Image Suite with adaptive aspect ratio was developed for fair image-to-video evaluation.

Main Results:

• The benchmark provides fine-grained metrics to reveal model-specific strengths and weaknesses.
• Analysis offers insights into current model capabilities across diverse content types and generation tasks.
• VBench++ facilitates comparison between video and image generation models.

Conclusions:

• VBench++ offers a holistic approach to video generation model evaluation, addressing limitations of existing metrics.
• The benchmark's open-source nature and leaderboard aim to accelerate progress in the field.
• VBench++ provides valuable insights for developing more human-aligned and trustworthy AI video generation systems.