BERTtoCNN: Similarity-preserving enhanced knowledge distillation for stance detection

Area of Science:

• Natural Language Processing
• Artificial Intelligence

Background:

• Stance detection, crucial for sentiment analysis, aims to identify authorial stance (favor/against) towards a target.
• Pre-trained models like BERT excel at stance detection but are computationally expensive for resource-limited scenarios.

Purpose of the Study:

• To develop an efficient and high-performing model for stance detection.
• To address the computational cost limitations of large pre-trained language models in real-world applications.

Main Methods:

• Proposes BERTtoCNN, a novel knowledge distillation framework combining classic distillation and similarity-preserving losses.
• Trains a compact CNN 'student' model from a large BERT 'teacher' model.
• Utilizes similarity-preserving loss to ensure consistent activation patterns between teacher and student networks.

Main Results:

• The BERTtoCNN model demonstrates superior performance compared to competitive baseline methods.
• Experiments conducted on open Chinese and English stance detection datasets validate the model's effectiveness.
• Achieves improved efficiency without compromising detection accuracy.

Conclusions:

• BERTtoCNN offers an efficient solution for stance detection, making advanced NLP techniques more accessible.
• The proposed knowledge distillation approach effectively transfers knowledge from large models to smaller, more efficient architectures.
• This work advances the practical application of stance detection in resource-constrained environments.