Vision-Language Models for automated quality control: a benchmarking framework and comprehensive study.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Automated object detection systems need quality assessment for real-world deployment.
• Traditional methods lack semantic understanding for effective model adaptation.
• Vision-Language Models (VLMs) offer potential for semantic-level quality control.

Purpose of the Study:

• To comprehensively benchmark Vision-Language Models (VLMs) for semantic quality assessment of object detection.
• To evaluate VLM performance across diverse domains including medical imaging, aerial surveillance, and industrial inspection.
• To provide evidence-based guidelines for VLM deployment in computer vision systems.

Main Methods:

• Systematic evaluation of nine state-of-the-art VLMs on five diverse object detection datasets.
• Rigorous statistical analysis using multi-class classification, accuracy metrics, coefficient of variation, and Kruskal-Wallis testing.
• Development of operational tiers based on deployment criticality and performance metrics.

Main Results:

• Substantial performance heterogeneity observed across VLMs and domains (accuracy 8.5%-82.8%).
• LLaVA-13B achieved the highest overall accuracy (48.6%), while medical domains presented the greatest challenge.
• Significant inter-model performance differences confirmed across all domains (p<0.001).

Conclusions:

• Current VLMs demonstrate suitability for supervised deployment in quality control, not fully autonomous operation.
• Established operational tiers guide VLM implementation based on domain criticality.
• The benchmark provides crucial insights and open-source tools for VLM-based quality assessment in production computer vision.