Zero-Shot Neural Network Evaluation with Sample-Wise Activation Patterns | JoVE Visualize

Area of Science:

Artificial Intelligence
Machine Learning
Computer Vision
Natural Language Processing

Background:

Zero-shot proxies are crucial for efficient neural network evaluation, particularly in Neural Architecture Search (NAS).
Existing zero-shot metrics suffer from limited generalization across different network architectures (CNNs, Transformers) and downstream tasks, along with weak performance correlation.
There is a need for a universal, training-free metric that accurately predicts neural network performance across diverse domains.

Purpose of the Study:

To introduce SWAP-Score, a novel, universal zero-shot metric for neural network evaluation.
To demonstrate SWAP-Score's broad applicability across different architecture families and task domains.
To establish SWAP-Score as a superior alternative to existing zero-shot metrics in terms of predictive performance and generalization.

Main Methods:

Propose Sample-Wise Activation Patterns (SWAP) to measure neural network expressivity over mini-batches.
Develop SWAP-Score, a derivative metric based on SWAP, for zero-shot evaluation.
Validate SWAP-Score's performance against existing metrics on computer vision and natural language processing tasks using CNNs and Transformers.

Main Results:

SWAP-Score exhibits strong correlations with ground-truth performance, achieving a Spearman's correlation of 0.93 for CNNs on CIFAR-10 and 0.71 for Transformers on GLUE tasks.
The metric demonstrates broad applicability across architecture families (CNNs, Transformers) and task domains (vision, NLP).
SWAP-empowered NAS (SWAP-NAS) achieves competitive performance with significantly reduced computational cost (e.g., ~6 minutes on CIFAR-10).

Conclusions:

SWAP-Score is a highly effective and broadly applicable zero-shot metric that overcomes limitations of existing methods.
Its label-independent nature allows application during pre-training for performance estimation.
SWAP-Score enables efficient and competitive Neural Architecture Search, significantly reducing evaluation time.