StochCA: A novel approach for exploiting pretrained models with cross-attention

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Large-scale pretrained models are crucial for improving performance on downstream tasks.
• Standard fine-tuning may not fully exploit the knowledge within these pretrained models.
• Transformer architectures are widely used but require effective fine-tuning strategies.

Purpose of the Study:

• Introduce a novel fine-tuning method, stochastic cross-attention (StochCA), for Transformer models.
• Enable selective utilization of knowledge from pretrained models during fine-tuning.
• Improve performance in transfer learning and domain generalization tasks.

Main Methods:

• Modified the Transformer's self-attention mechanism to incorporate cross-attention.
• Stochastically performed cross-attention using keys and values from a pretrained model's corresponding block.
• Fine-tuned queries and channel-mixing layers of the target model to exploit pretrained representations.

Main Results:

• Stochastic cross-attention (StochCA) demonstrated superior performance over state-of-the-art methods.
• Achieved significant improvements in both transfer learning and domain generalization benchmarks.
• Showed that StochCA is complementary to existing fine-tuning approaches and can be combined for further gains.

Conclusions:

• StochCA effectively enhances knowledge exploitation from pretrained models in Transformer architectures.
• The proposed method offers a significant advancement for fine-tuning strategies.
• StochCA provides a flexible and effective approach for improving model performance across various tasks.