Exploring transformer models: Fine-tuning VS inference on relation extraction from biomedical texts

Area of Science:

• Biomedical informatics
• Artificial Intelligence
• Natural Language Processing

Background:

• Biomedical data volume is rapidly increasing, making manual information extraction infeasible.
• Biomedical relation extraction automates discovering relationships in text, crucial for knowledge discovery.
• Existing Transformer models require costly, expert-created datasets for fine-tuning.

Purpose of the Study:

• To evaluate the reliability of Generative Artificial Intelligence (GenAI) models for biomedical relation extraction.
• To compare the performance of fine-tuned Transformer models against various generative Large Language Models (LLMs).
• To assess the impact of domain-specific pretraining on generative LLM performance.

Main Methods:

• Compared fine-tuned Transformer models (T5, PubMedBERT, etc.) with generative LLMs (Mistral-7B, LLaMA2-7B, LLaMA3-8B, Gemma, RAG, Me-LLaMA-13B).
• Evaluated models on four key biomedical relation extraction tasks: chemical-protein, disease-protein, drug-drug interaction, and protein-protein interaction.
• Utilized identical datasets for both fine-tuned and generative model experiments.

Main Results:

• Fine-tuned Transformer models achieved significantly higher scores (84.42-90.35) compared to generative LLMs (36.64-53.94).
• Generative LLMs performance was roughly half that of fine-tuned models.
• Me-LLaMA, pretrained on MIMIC-III, showed improved performance (45.76) over general-domain pretrained models, demonstrating the value of specialized pretraining.

Conclusions:

• Fine-tuned Transformer models remain superior for biomedical relation extraction tasks.
• Generative LLMs show potential but require substantial domain-specific pretraining for competitive performance.
• Domain-specific pretraining is critical for enhancing the effectiveness of generative models in specialized fields like biomedicine.