SEA: Hierarchically searching efficient adapters for pre-trained models

Adapting large-scale pre-trained models for downstream tasks has proven to be effective in improving performance. However, relying on the hand-crafted adapters may limit generalizability and incur extra inference overhead. Existing neural architecture search (NAS) techniques for designing adapters generally focus on hyper-parameters, neglecting the design of the search space and the biased problem in the supernet training. To address these limitations, this paper proposes a general framework to Search Efficient Adapters (SEA), providing a hierarchical adapter search space from a systematic perspective and introducing a sample-efficient supernet training mechanism. Specifically, SEA searches for efficient adapters at various levels of granularity, enabling refined adapter depths, widths, and channel combinations. The hierarchical search space encompasses a diverse range of adapter structures, enabling generalizability across multiple tasks. Furthermore, the searched adapters can be seamlessly integrated into the pre-trained models, eliminating any additional inference overhead. To avoid training bias of different channels during the width search, we implement a sample-efficient training mechanism inspired by the Upper Bound Confidence (UCB) theory. This mechanism promotes equal opportunities for training channels at early stages and prioritizes training of channels with larger gradient norms later, enhancing the training's effectiveness. We provide adaption results on a wide range of tasks, showing the superiority of our method. SEA achieves 0.53% higher average accuracy than GLoRA across 19 few-shot tasks on VTAB-1k. When adapted on COCO, SEA boosts mean average precision by 7.3% (detection) and 5.6% (segmentation) over SSF. Moreover, we delve into the relationship between adapter structures and downstream tasks, unveiling interesting phenomena that may offer insights for future adapter design. Our code is available at https://github.com/ShunLu91/SEA.