Optimised knowledge distillation for efficient social media emotion recognition using DistilBERT and ALBERT.

Area of Science:

• Natural Language Processing
• Machine Learning
• Affective Computing

Background:

• Accurate emotion recognition in social media text is vital for various applications but faces challenges like computational complexity and class imbalance.
• Transformer models offer high performance but are too large and slow for real-time, resource-constrained environments.

Purpose of the Study:

• To propose a novel knowledge distillation framework for efficient emotion recognition.
• To transfer knowledge from a large BERT-base model to smaller DistilBERT and ALBERT models.

Main Methods:

• Implemented a knowledge distillation framework with a hybrid loss function (focal loss and KL divergence) to improve minority class recognition.
• Utilized attention-head alignment for effective knowledge transfer and semantic-preserving data augmentation to address class imbalance.
• Trained and evaluated models on two large-scale social media emotion datasets (Twitter Emotions and Social Media Emotion).

Main Results:

• Distilled models achieved near-teacher performance with minimal accuracy drop (<1% and <6%).
• Model size was reduced by 40%, and inference latency decreased by 3.2×.
• Significantly improved F1-scores for minority emotion classes.

Conclusions:

• The proposed knowledge distillation framework enables efficient and accurate emotion recognition.
• This approach overcomes the limitations of large transformer models, facilitating deployment in edge computing and mobile applications.
• Sets a new state-of-the-art for efficient emotion recognition in social media text.