Publication Type Tagging using Transformer Models and Multi-Label Classification.

Area of Science:

• Biomedical Informatics
• Natural Language Processing
• Computational Linguistics

Background:

• Efficiently searching and filtering the biomedical literature requires indexing by publication type and study design.
• This type of indexing is less studied than topical term indexing (e.g., MeSH).

Purpose of the Study:

• To develop and evaluate state-of-the-art Transformer-based models for automatic tagging of publication types and study designs.
• To leverage human-curated PubMed data for training.

Main Methods:

• Generated a dataset of over 1.2 million article titles and abstracts from PubMed.
• Trained PubMedBERT-based models using a multi-label classification approach.
• Explored techniques like undersampling, feature verbalization, and contrastive learning to enhance performance.

Main Results:

• PubMedBERT demonstrated a strong baseline for indexing publication types and study designs.
• Undersampling, feature verbalization, and unsupervised contrastive loss positively impacted performance.
• The best model, utilizing 80% undersampling and feature verbalization, achieved a macro-F1 of 0.632 and macro-AUC of 0.969, outperforming previous models (MultiTagger).

Conclusions:

• The developed PubMedBERT-based model offers a significant improvement for automated indexing of publication types and study designs in biomedical literature.
• Future work could incorporate full-text features and explore model interpretability.