InfoMSD: an information-maximization self-distillation framework for parameter-efficient fine-tuning on artwork...

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Large-scale vision-language models excel at visual tasks but are computationally expensive.
• Resource-constrained environments like cultural heritage settings face deployment challenges.
• Object recognition in artwork requires identifying depicted items, distinct from style classification.

Purpose of the Study:

• To develop an unsupervised, parameter-efficient framework for object recognition in artwork.
• To address the high computational costs and parameter counts of existing models.
• To enable robust performance in resource-limited cultural and artwork settings.

Main Methods:

• Proposed InfoMSD, an Information-Maximization Self-Distillation framework.
• Utilized a teacher-student architecture for generating pseudo-labels and learning via cross-entropy.
• Implemented parameter-efficient fine-tuning by updating only layer norm and visual prompts, freezing other parameters.
• Applied entropy-based regularization for sharpening probability distributions and balancing class coverage.

Main Results:

• InfoMSD achieved accuracy improvements of +6.43% and +3.02% over CLIP zero-shot baselines.
• The framework updated less than 1% of model parameters, demonstrating significant efficiency.
• Compared to lightweight distillation methods, InfoMSD showed average accuracy gains of 1.35% and 0.96%.
• Enhanced pseudo-label quality and model discriminative capacity were observed.

Conclusions:

• InfoMSD offers a novel, information-theoretic approach for unsupervised and efficient fine-tuning in artwork object recognition.
• The framework effectively balances high performance with reduced computational overhead.
• InfoMSD is well-suited for deployment in resource-constrained cultural and artwork domains.