Toward Transparent Deep Image Aesthetics Assessment With Tag-Based Content Descriptors

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Deep learning models for Image Aesthetics Assessment (IAA) lack transparency, with internal mechanisms and learned semantic features remaining unclear.
• Existing methods often rely on implicitly learned semantic features, hindering a full understanding of what contributes to aesthetic predictions.

Purpose of the Study:

• To develop a more transparent deep learning framework for IAA by introducing explainable semantic features.
• To enhance the interpretability of IAA models by utilizing human-readable content descriptions.

Main Methods:

• Proposed Tag-based Content Descriptors (TCDs) to quantify image relevance to human-readable tags.
• Introduced an explicit matching process using predefined tags for TCD generation.
• Developed an implicit matching process to capture image content not covered by predefined tags, integrated into TCDs.
• Utilized a TCD generator to realize both explicit and implicit matching processes.

Main Results:

• An IAA model using TCDs based on predefined tags achieved a Spearman Rank Correlation Coefficient (SRCC) of 0.767, comparable to state-of-the-art methods.
• Integrating implicit matching components into TCDs boosted the IAA model's performance to an SRCC of 0.817, outperforming existing IAA methods.
• The TCD generator demonstrated effective assignment of photography-related tags to images on a custom validation set.

Conclusions:

• Tag-based Content Descriptors offer a transparent and effective approach to Image Aesthetics Assessment.
• The proposed explicit and implicit matching processes within TCDs significantly enhance the accuracy and interpretability of IAA models.
• The TCD generator provides a robust mechanism for describing image content semantically, advancing the field of explainable AI in image analysis.