DTL: Parameter- and Memory-Efficient Disentangled Vision Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Finetuning large pretrained models incurs significant computational costs.
• Existing parameter-efficient transfer learning (PETL) methods struggle to reduce GPU memory footprint during training due to parameter entanglement.
• The need for memory-efficient and accurate transfer learning techniques is critical for large-scale AI models.

Purpose of the Study:

• To introduce Disentangled Transfer Learning (DTL) to reduce GPU memory usage and trainable parameters in large model finetuning.
• To propose a novel Compact Side Network (CSN) architecture for effective knowledge transfer.
• To demonstrate the superiority of DTL over existing PETL methods in terms of both efficiency and accuracy.

Main Methods:

• DTL disentangles trainable parameters from the backbone model using a lightweight Compact Side Network (CSN).
• CSN employs low-rank linear mappings to extract and re-inject task-specific information.
• DTL is extended with sparse architectural designs for complex tasks like object detection and semantic segmentation.

Main Results:

• DTL significantly reduces GPU memory usage and the number of trainable parameters compared to existing PETL methods.
• The proposed CSN effectively facilitates knowledge transfer across various downstream recognition tasks.
• DTL achieves superior accuracy margins over current PETL approaches on benchmark datasets.

Conclusions:

• DTL offers a more efficient and effective approach to transfer learning for large models.
• The CSN architecture provides a viable solution for memory-constrained training scenarios.
• DTL represents a significant advancement in parameter-efficient transfer learning, particularly for computer vision tasks.