CLIP knows image aesthetics

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Image Aesthetic Assessment (IAA) commonly uses ImageNet classification models for pretraining.
• This pretraining may not be optimal for IAA as it discourages learning features like composition and style.

Purpose of the Study:

• To investigate if Contrastive Language-Image Pretraining (CLIP) is a superior base for IAA models compared to ImageNet models.
• To evaluate CLIP's effectiveness in extracting features relevant to image aesthetics.

Main Methods:

• Engineered natural language prompts for CLIP to assess image aesthetics without weight adjustments.
• Developed a strategy to convert CLIP's prompt-based assessments into a continuous scalar score.
• Trained linear regression models on the AVA dataset using features from CLIP and ImageNet models.
• Fine-tuned CLIP's image encoder on the AVA dataset.

Main Results:

• CLIP-based models, even with linear regression, outperformed ImageNet-based linear regression models.
• Fine-tuning CLIP's image encoder required fewer epochs and achieved better performance than fine-tuned ImageNet models.
• CLIP demonstrated superior feature extraction for IAA tasks.

Conclusions:

• CLIP is a more suitable base model for Image Aesthetic Assessment than traditional ImageNet pretrained networks.
• CLIP's natural language supervision enables learning of diverse image features beneficial for aesthetic evaluation.