Attention-Based Models for Classifying Small Data Sets Using Community-Engaged Research Protocols: Classification...

Area of Science:

• Computational Social Science
• Health Informatics
• Artificial Intelligence

Background:

• Community-Engaged Research (CEnR) partners scholars with communities for mutual benefit.
• Universities face challenges in reporting CEnR metrics, hindering infrastructure development.
• Accurate CEnR metrics are crucial for community relationships, funding, and stakeholder engagement.

Purpose of the Study:

• To develop and evaluate deep learning models for identifying and categorizing community-engaged research (CEnR) in human participant protocols.
• To apply attention-based deep learning and transformer models to institutional review board (IRB) submissions.

Main Methods:

• Manually classified 280 IRB protocols using a 3- and 6-level CEnR heuristic.
• Trained an attention-based Bi-LSTM and compared it with transformer models (BERT, Bio+Clinical BERT, XLM-RoBERTa).
• Applied best-performing models to over 6000 unlabeled IRB protocols (2013-2019).

Main Results:

• Transformer models achieved a 0.9952 F1 score, significantly outperforming Bi-LSTM (48%-80%).
• Overfitting was a key issue across various methodological adjustments and dataset configurations.
• Despite overfitting, transformer models demonstrated a superior understanding of CEnR data compared to Bi-LSTM.

Conclusions:

• Transfer learning methods, particularly transformer models like BERT, are more effective than attention-based Bi-LSTM for analyzing CEnR descriptions in research protocols.
• These models offer a promising approach to address the real-world challenge of CEnR metric reporting.
• Further refinement is needed to overcome overfitting and fully leverage deep learning for CEnR analysis.