Training-Free Transformer Architecture Search With Zero-Cost Proxy Guided Evolution.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Designing high-performance Transformer architectures is complex and time-consuming.
• Existing training-free proxy methods for Neural Architecture Search (NAS) show poor generalization in Transformer search spaces.
• There is a need for efficient and effective Transformer Architecture Search (TAS) methods.

Purpose of the Study:

• To develop an efficient training-free method for Transformer Architecture Search (TAS).
• To introduce a novel evaluation metric that correlates with Transformer performance.
• To improve the efficiency and effectiveness of automated Transformer architecture design.

Main Methods:

• Introduced TRansformer Architecture search with ZerO-cost pRoxy guided evolution (T-Razor).
• Proposed DSS++ metric based on synaptic diversity of multi-head self-attention (MSA) and saliency of multi-layer perceptron (MLP).
• Employed block-wise evolution search guided by DSS++ for optimizing Transformer architectures.

Main Results:

• T-Razor achieved competitive performance against state-of-the-art Transformer architectures across four search spaces.
• Demonstrated significant improvements in search efficiency, reducing GPU days from over 24 to under 0.4.
• Searched Transformers using T-Razor achieved competitive results on GLUE benchmark for NLP tasks.

Conclusions:

• T-Razor offers a highly efficient and effective solution for training-free Transformer Architecture Search (TAS).
• The DSS++ metric effectively evaluates Transformers based on block-level properties, enhancing search capabilities.
• This work provides valuable insights into training-free TAS and automated Transformer design.