Zero-Shot Neural Network Evaluation With Sample-Wise Activation Patterns

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision
• Natural Language Processing

Background:

• Zero-shot proxies are crucial for efficient neural network evaluation, particularly in Neural Architecture Search (NAS).
• Existing zero-shot metrics suffer from limited generalization across different network architectures (CNNs, Transformers) and downstream tasks, along with weak performance correlation.
• There is a need for a universal, training-free metric that accurately predicts neural network performance across diverse domains.

Purpose of the Study:

• To introduce SWAP-Score, a novel, universal zero-shot metric for neural network evaluation.
• To demonstrate SWAP-Score's broad applicability across different architecture families and task domains.
• To establish SWAP-Score as a superior alternative to existing zero-shot metrics in terms of predictive performance and generalization.

Main Methods:

• Propose Sample-Wise Activation Patterns (SWAP) to measure neural network expressivity over mini-batches.
• Develop SWAP-Score, a derivative metric based on SWAP, for zero-shot evaluation.
• Validate SWAP-Score's performance against existing metrics on computer vision and natural language processing tasks using CNNs and Transformers.

Main Results:

• SWAP-Score exhibits strong correlations with ground-truth performance, achieving a Spearman's correlation of 0.93 for CNNs on CIFAR-10 and 0.71 for Transformers on GLUE tasks.
• The metric demonstrates broad applicability across architecture families (CNNs, Transformers) and task domains (vision, NLP).
• SWAP-empowered NAS (SWAP-NAS) achieves competitive performance with significantly reduced computational cost (e.g., ~6 minutes on CIFAR-10).

Conclusions:

• SWAP-Score is a highly effective and broadly applicable zero-shot metric that overcomes limitations of existing methods.
• Its label-independent nature allows application during pre-training for performance estimation.
• SWAP-Score enables efficient and competitive Neural Architecture Search, significantly reducing evaluation time.