A Unified Masked Jigsaw Puzzle Framework for Vision and Language Models

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Federated learning (FL) utilizes Transformer architectures for Computer Vision (CV) and Natural Language Processing (NLP) tasks.
• Transformers are vulnerable to gradient attacks due to sensitive information in Position Embeddings (PEs).
• PEs can be exploited to reconstruct input data, posing a security risk in FL.

Purpose of the Study:

• To introduce a novel framework, Masked Jigsaw Puzzle (MJP), to enhance Transformer robustness against gradient attacks in FL.
• To improve the performance of Transformer models in both CV and NLP applications.
• To develop a unified approach for securing Transformer-based models across different domains.

Main Methods:

• The Masked Jigsaw Puzzle (MJP) framework employs random token shuffling to disrupt positional order.
• A learnable unknown (unk) position embedding is utilized to mask the PEs of shuffled tokens.
• This process disrupts local spatial information, forcing models to learn more robust representations.

Main Results:

• MJP significantly improves the robustness of Transformer models against gradient-based attacks.
• The framework boosts model performance in image classification (ImageNet-1K) and text sentiment analysis (Yelp, Amazon).
• Experimental results validate MJP as an effective unified framework for both vision and language tasks.

Conclusions:

• The Masked Jigsaw Puzzle (MJP) framework effectively mitigates gradient attacks in federated learning.
• MJP enhances Transformer performance across diverse CV and NLP applications.
• MJP offers a unified and robust solution for securing Transformer models in federated settings.