Q-Eval+: From Absolute to Relative Quality Evaluation for Text-to-Vision Content

Text-to-vision (T2V) models are rapidly reshaping content creation, yet their outputs still require careful human screening before deployment in real applications. This creates an urgent need for evaluation tools that can 1) assign reliable absolute quality scores and 2) support fine-grained relative comparisons for model selection, reranking, and online A/B testing. Existing T2V benchmarks, however, are often limited by entangled definitions of visual quality and text-vision alignment, narrow coverage of images or videos, and the absence of systematically constructed pairwise and diagnostic annotations. Therefore, we present Q-Eval+, a large-scale dataset to jointly support absolute scoring and relative comparing for T2V content. Q-Eval+ consists of 220K image/video instances with 1.32M human annotations along two key dimensions: Q-Eval-Abs provides MOS-style labels for visual quality and alignment at both instance and model levels, while Q-Eval-Rel offers side-by-side preferences and choice-based VQA questions that reveal not only which instance is better, but also why. On top of Q-Eval+, we develop Q-Evaluator, a unified large multimodal model that bridges absolute and relative assessment through supervised fine-tuning with structured context prompts and a Vague-to-Specific alignment handling strategy. We further introduce symmetric and consistency accuracies to evaluate robustness under pairwise permutations. Extensive experiments show that Q-Evaluator achieves consistently strong correlations with human opinions and competitive comparative accuracy against existing open-source and proprietary LMM-based evaluators, providing a practical foundation for robust T2V quality control.