Syntax-Oriented Shortcut: A Syntax Level Perturbing Algorithm for Preventing Text Data From Being Learned.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning Security

Background:

• Large language models (LLMs) rely heavily on vast amounts of publicly available data.
• Concerns exist regarding the unauthorized use of this data for LLM training.
• Existing data protection methods for images are unsuitable for text due to semantic changes.

Purpose of the Study:

• To propose a novel algorithm for generating unlearnable text examples.
• To protect text data from unauthorized use in LLM training.
• To address the limitations of applying image-based perturbation methods to text.

Main Methods:

• Developed Unlearnable text examples generation algorithm via syntax-oriented shortcut (UTE-SS).
• Introduced a syntax template generator (STG) for optimal category-specific syntax perturbation.
• Designed a perturbing text generator (PTG) to modify texts using syntax templates, creating imperceptible yet effective deviations.

Main Results:

• The UTE-SS algorithm successfully generates unlearnable text examples.
• Models are misled to learn syntax-category shortcuts, preventing information mining.
• Demonstrated effectiveness and flexibility across eight Transformer-based pretrained language models (PLMs) and four NLP tasks.

Conclusions:

• The proposed UTE-SS method offers an effective solution for protecting text data from unauthorized LLM training.
• The syntax-oriented approach overcomes challenges in perturbing discrete text data.
• The algorithm is easy to implement and has publicly available code.