Do LLMs Surpass Encoders for Biomedical NER?

Area of Science:

• Biomedical Informatics
• Natural Language Processing
• Computational Linguistics

Background:

• Biomedical Named Entity Recognition (NER) is crucial for information extraction.
• Transformer-based encoder models (e.g., BERT) are the current standard for NER.
• Large Language Models (LLMs) are emerging in information extraction but may ignore positional information and are computationally expensive.

Purpose of the Study:

• To evaluate the performance and efficiency of LLMs for biomedical NER compared to encoder models.
• To assess the trade-offs between performance gains and computational costs of LLMs.
• To investigate the impact of entity length on NER performance using the BIO tagging scheme.

Main Methods:

• Utilized five diverse biomedical datasets with varying proportions of longer entities.
• Employed the BIO (Beginning, Inside, Outside) entity tagging scheme to retain positional information.
• Compared LLMs (Mistral, Llama 8B) against state-of-the-art encoder models (BERT, BiomedBERT, DeBERTav3 300M).

Main Results:

• LLMs outperformed encoder models by 2-8% in F-scores on most datasets.
• Performance gains were more significant for longer entities (≥ 3 tokens).
• LLMs exhibited inference times one to two orders of magnitude higher than encoder models.

Conclusions:

• LLMs offer superior performance in biomedical NER, particularly for complex entities.
• Encoder models may be more suitable when computational efficiency and real-time processing are critical.
• The choice between LLMs and encoder models depends on specific application requirements and resource constraints.