InA: Inhibition Adaption on pre-trained language models.

Area of Science:

• Natural Language Processing
• Machine Learning

Background:

• Pre-trained language models (LMs) require adaptation for specific downstream tasks.
• Existing adaptation fine-tuning methods need clearer mechanisms and better control over information transmission.

Purpose of the Study:

• To introduce a novel fine-tuning method, Inhibition Adaptation (InA), for pre-trained LMs.
• To reduce the number of tunable weights and reweight knowledge from pre-trained LMs.

Main Methods:

• InA inserts trainable vectors into Transformer attention architectures.
• A threshold mechanism is employed to eliminate irrelevant knowledge, inspired by shunting inhibition.

Main Results:

• InA achieves competitive or superior performance compared to existing fine-tuning methods.
• The method was evaluated on BERT-large, RoBERTa-large, and DeBERTa-large models.
• Effective for text classification and question-answering tasks.

Conclusions:

• Inhibition Adaptation (InA) offers an efficient and effective approach to fine-tuning language models.
• The method provides a mechanism to control knowledge transfer, improving task-specific performance.