ScaleNet: Scaling up Pretrained Neural Networks With Incremental Parameters.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Deep Learning

Background:

• Larger Vision Transformers (ViTs) demonstrate superior performance but require extensive computational resources for training.
• Scaling ViT models presents a significant challenge due to high training costs and time.

Purpose of the Study:

• To introduce ScaleNet, an efficient method for expanding ViT models.
• To enable rapid and cost-effective scaling of ViTs by leveraging pretrained models.

Main Methods:

• ScaleNet inserts additional layers into pretrained ViTs, employing layer-wise weight sharing for parameter efficiency.
• Adjustment parameters are introduced via parallel adapter modules to optimize shared weights and prevent performance degradation.

Main Results:

• ScaleNet facilitates efficient expansion of ViT models, as demonstrated on the ImageNet-1K dataset.
• A $2\times$ depth-scaled DeiT-Base model using ScaleNet achieved a 7.42% accuracy improvement over training from scratch.
• The method required only one-third of the training epochs compared to traditional training from scratch.

Conclusions:

• ScaleNet offers a cost-effective solution for scaling ViT models, enhancing performance with minimal parameter increase.
• The approach shows promise for downstream vision tasks, including object detection, beyond image classification.