Multi-Modality Multi-Attribute Contrastive Pre-Training for Image Aesthetics Computing.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Current Image Aesthetics Computing (IAC) methods often rely on ImageNet pre-trained backbones.
• These backbones prioritize object semantics, neglecting high-level aesthetic concepts, leading to suboptimal performance in IAC tasks.

Purpose of the Study:

• To develop an alternative pre-training framework for Image Aesthetics Computing (IAC) that surpasses ImageNet-based approaches.
• To address the limitations of existing methods in capturing nuanced image aesthetics.

Main Methods:

• A multi-modality, multi-attribute contrastive pre-training framework was proposed.
• A multi-attribute image description database was created using human feedback and multi-modality large language models.
• Visual and text features were integrated and mapped to embedding spaces for multi-attribute contrastive learning.
• A semantic affinity loss was introduced to mitigate domain shift and improve generalization.

Main Results:

• The proposed framework achieved state-of-the-art performance on Image Aesthetics Computing (IAC) tasks.
• The integration of visual and textual features led to a more comprehensive aesthetic representation.
• The semantic affinity loss effectively improved model generalization across domains.

Conclusions:

• The novel pre-training framework offers a superior alternative to ImageNet-based pre-training for IAC.
• This approach enhances the ability of models to understand and compute image aesthetics.
• The findings pave the way for more sophisticated aesthetic analysis in computer vision.