Exploring Themes and Bias in Art using Machine Learning Image Analysis

Area of Science:

• Computer Vision
• Artificial Intelligence
• Art History

Background:

• Museums are increasingly providing open access collections.
• Manual art object annotation is laborious and time-consuming.
• Enhancing user experience with accurately labeled art is crucial for open collections.

Purpose of the Study:

• To apply machine learning for automated classification of art images from The Met's open access collection.
• To compare the performance of ResNet 50, ResNet 101, and Inception-ResNet-V2 models.
• To enhance model interpretability using Gradient-weighted Class Activation Maps (Grad-CAMs) and explore potential biases in gender labels.

Main Methods:

• Convolutional Neural Networks (CNNs): ResNet 50, ResNet 101, Inception-ResNet-V2.
• Gradient-weighted Class Activation Maps (Grad-CAMs) for model interpretability.
• Multi-label classification implementation using ResNet 50.

Main Results:

• Comparative analysis of three deep learning models for art image classification.
• Identification of potential biases in gender labels through interpretable AI techniques.
• Successful implementation of a multi-label classification model.

Conclusions:

• Deep learning offers a viable alternative to manual annotation for large-scale art collections.
• Interpretable AI methods like Grad-CAMs are essential for understanding model behavior and bias.
• Automated art image classification can significantly improve user engagement with open museum collections.