Larger models yield better results? Streamlined severity classification of ADHD-related concerns using BERT-based...

Area of Science:

• Natural Language Processing (NLP)
• Machine Learning
• Computational Linguistics

Background:

• BERT-based models offer powerful NLP capabilities but are computationally intensive.
• Knowledge distillation is a technique to create smaller, efficient models from larger ones.
• Classifying mental health concerns from social media requires accurate and accessible NLP tools.

Purpose of the Study:

• To develop a lightweight BERT-based model using knowledge distillation for NLP applications.
• To evaluate the efficiency and performance of the resulting model, LastBERT, on a real-world task.
• To assess LastBERT's utility in classifying Attention Deficit Hyperactivity Disorder (ADHD) severity from social media text.

Main Methods:

• Implemented knowledge distillation to create a customized student BERT model (LastBERT).
• Reduced model parameters from 110 million (BERT-base) to 29 million.
• Evaluated LastBERT on the General Language Understanding Evaluation (GLUE) benchmark and a real-world ADHD dataset.

Main Results:

• LastBERT achieved a 73.64% reduction in model size compared to BERT-base.
• The model demonstrated strong performance on GLUE tasks.
• On the ADHD dataset, LastBERT achieved 85% accuracy, F1 score, precision, and recall.
• LastBERT showed comparable performance to DistilBERT and ClinicalBERT.

Conclusions:

• Knowledge distillation can produce effective, lightweight NLP models suitable for resource-limited environments.
• LastBERT is a viable tool for mental health professionals to analyze social media data for ADHD severity.
• The study highlights the accessibility and practicality of advanced NLP methods in real-world applications.