Covariance-Inverse Fusion: Toward Comprehensive Proxies for Zero-Shot Neural Architecture Search

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Science

Background:

• Zero-cost proxy fusion methods for Neural Architecture Search (NAS) offer efficiency but require extensive ground-truth data.
• Current methods face high computational costs due to the need for proxy generation and validation.
• Automated fusion mechanisms are crucial for integrating diverse zero-cost proxies effectively.

Purpose of the Study:

• To develop an automated fusion mechanism for zero-cost proxies in NAS.
• To reduce the dependency on ground-truth samples and computational overhead in architecture evaluation.
• To propose a novel fusion method that leverages complementary strengths of existing proxies.

Main Methods:

• Introduced the Covariance-Inverse Fusion (CoIF) method, encoding complementary information in the inverse covariance matrix.
• Utilized covariance-weighted fusion to leverage strengths of diverse zero-cost proxies.
• Theoretically proved optimal fusion weights and nondecreasing consistency, with low ground-truth dependency.

Main Results:

• CoIF achieved competitive performance using 80% fewer samples and 99% less fusion time than existing learning-based methods.
• Experiments on CNNs, vision transformers, and LLMs demonstrated CoIF's ability to integrate diverse architectural properties.
• The method ensures efficiency and theoretical optimality in linear fusion cases.

Conclusions:

• CoIF provides an effective approach for zero-shot NAS by optimizing proxy fusion.
• The method significantly reduces computational costs and data requirements.
• CoIF establishes a new standard for efficient and reliable architecture evaluation in NAS.