Vaner2: towards more general biomedical named entity recognition using multi-task large language model encoders

Biomedical named entity recognition (BioNER) serves as the foundation for many downstream tasks, such as relation extraction, question answering, and clinical text analysis. BioNER was previously dominated by BERT-based models pretrained on large biomedical corpora. However, BERT-based models finetuned on specific BioNER datasets exhibit limited ability to generalize to other datasets. With the recent advances of large language models (LLMs), several works have fine-tuned autoregressive LLMs that are not inherently suitable for BioNER tasks, which limits model performance. In this study, building upon our previous work VANER, we utilized LLMs with the causal attention mask removed as a text encoder for sequence labeling. Using 39 BioNER datasets, we trained a multi-task NER model that extracts all entity types with one LLM forward pass. We also proposed a token-wise loss rescaling technique to deal with the data imbalance between different tags and entity types. Extensive experiments on independent test datasets demonstrated that our VANER2 model achieved the best generalization results compared with BERT-based baselines and several recent BioNER methods. VANER2 is freely available at https://github.com/ZhuLab-Fudan/VANER2.