Benchmarking and Analyzing Generative Data for Visual Recognition.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Large pre-trained generative models show promise for data generation in visual recognition.
• Evaluating the efficacy of generative data against traditional methods like retrieval or original data is crucial.

Purpose of the Study:

• To comprehensively benchmark generative data's impact on visual recognition tasks.
• To introduce a novel metric (CLER score) for assessing generative data efficiency.
• To explore methods for enhancing generative data utility through external knowledge injection.

Main Methods:

• Construction of GenBench, a benchmark with 22 datasets and 2548 categories.
• Proposal of the CLER (Classification Efficiency Rank) score, a training-free metric.
• Comparative analysis of generative, retrieved, and original data paradigms.
• Application of Textual Inversion for fine-tuning token embeddings to inject external knowledge.

Main Results:

• GenBench provides a standardized evaluation framework for generative data.
• The CLER score demonstrates a better correlation with downstream recognition performance than existing metrics.
• Generative data exhibits unique characteristics compared to retrieved data.
• Fine-tuning with Textual Inversion improved performance on 17 out of 22 datasets, particularly with high-resolution images.

Conclusions:

• Generative data holds significant potential for advancing visual recognition.
• The CLER score offers a valuable tool for pre-training evaluation of generative datasets.
• Further research is needed to address challenges, especially with low-resolution data.