DiSMix: Dimensional Swap Mix for Feature-Level Data Augmentation in Vision Transformers

Mixup is a data augmentation technique that improves prediction accuracy in classification tasks by combining representations of training samples, which makes it particularly effective in settings with limited data and during fine-tuning for downstream tasks. However, representations generated by mixup may appear unnatural, which can negatively affect fine-tuning performance. To address this limitation, we propose a vision transformer (ViT)-aware variant of mixup strategies, Dimensional Swap Mix (DiSMix). DiSMix divides a representation vector into two segments corresponding to subspaces of the original feature space and generates new representations by swapping one segment with that from another sample and concatenating the segments. This allows part of the original representation to remain unchanged, enabling the model to learn from partially preserved features. We evaluate DiSMix by applying several mixup-based methods to fine-tune ViTs on the VTAB-1k benchmark. The findings show that DiSMix improves accuracy on the VTAB-1k Natural split, reaching 80.0%, compared with conventional mixup methods. This suggests that DiSMix is an effective alternative for representation-level data augmentation in fine-tuning scenarios.