Distilling knowledge from multiple foundation models for zero-shot image classification.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Zero-shot image classification (ZIC) aims to recognize new categories without specific training data.
• Existing methods struggle with generalization when training data for all categories is unavailable.
• Foundation models offer powerful knowledge distillation capabilities for unseen classes.

Purpose of the Study:

• To develop a ZIC framework leveraging foundation models for enhanced generalization.
• To enable recognition of novel image categories absent from training datasets.
• To improve classification accuracy in data-scarce scenarios.

Main Methods:

• Utilized ChatGPT and DALL-E to synthesize reference images for unseen categories from text prompts.
• Employed CLIP and DINO for aligning test images with text and synthesized reference images.
• Calculated logits and aggregated predictions based on confidence for final classification.

Main Results:

• Achieved significant improvements in classification accuracy across multiple datasets (MNIST, SVHN, CIFAR-10/100, TinyImageNet).
• Attained Area Under the Receiver Operating Characteristic (AUROC) scores exceeding 96% on all tested datasets.
• Demonstrated superior performance compared to existing zero-shot image classification approaches.

Conclusions:

• The proposed framework effectively distills knowledge from foundation models for robust zero-shot image classification.
• This approach enhances model generalization capabilities for recognizing previously unseen image categories.
• The method provides a promising solution for real-world applications where labeled data is limited.