AMPLIFY: attention-based mixup for performance improvement and label smoothing in transformer

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Mixup is a data augmentation technique that combines samples to improve model generalization.
• Standard mixup methods can propagate noise and outliers from original samples to augmented data, negatively impacting model performance.
• Existing mixup techniques may require significant computational resources.

Purpose of the Study:

• To introduce AMPLIFY, a new mixup method designed to mitigate the propagation of noise and outliers.
• To enhance the robustness and performance of models trained with augmented data.
• To achieve these improvements with low computational overhead.

Main Methods:

• AMPLIFY utilizes the inherent attention mechanism of Transformers to selectively reduce the influence of noisy or aberrant features.
• The method integrates seamlessly with existing Transformer architectures without adding trainable parameters.
• It operates with minimal computational cost compared to other advanced mixup strategies.

Main Results:

• AMPLIFY demonstrated superior performance in text classification tasks across seven benchmark datasets.
• The proposed method effectively reduced model over-sensitivity to outliers and noisy samples.
• Achieved state-of-the-art results with significantly lower computational resource consumption.

Conclusions:

• AMPLIFY offers an effective and computationally efficient solution for noise-robust data augmentation in text classification.
• The integration of attention mechanisms provides a novel approach to enhance pre-trained models like BERT, ALBERT, RoBERTa, and GPT.
• This research opens new avenues for improving deep learning model performance through intelligent data augmentation.