Fine-tuning Strategies for Classifying Community-Engaged Research Studies Using Transformer-Based Models: Algorithm...

Area of Science:

• Computational Social Science
• Higher Education Research
• Public Health Research

Background:

• Community-engaged research (CEnR) is vital for university missions but challenging to track.
• Existing methods struggle with quantifying community involvement and reporting CEnR metrics.
• Accurate tracking is needed to understand the prevalence and impact of CEnR.

Purpose of the Study:

• To develop a novel method for classifying CEnR studies.
• To capture distinct levels of community partner involvement in research direction.
• To evaluate deep learning models, specifically transformer-based architectures, for CEnR classification.

Main Methods:

• Utilized fine-tuning strategies (discriminative learning rates, layer freezing) on transformer models (BERT, Bio+ClinicalBERT, XLM-RoBERTa).
• Trained and tested 135 modified classification models.
• Evaluated model generalizability using a holdout dataset.

Main Results:

• Bio+ClinicalBERT achieved 73.08% accuracy and 62.94% F1-score on the holdout dataset.
• All trained models outperformed previous methods by 10%-23% in accuracy and F1-score.
• Demonstrated the effectiveness of fine-tuning strategies for transformer models in CEnR classification.

Conclusions:

• Transfer learning and fine-tuning significantly improve transformer-based models for CEnR tracking.
• The developed tool aids in categorizing community engagement in research.
• Findings address challenges in reporting CEnR metrics and understanding community involvement.